Full Virtualization in Oracle Solaris 11: OS, Network, Storage
Introduction
Oracle Solaris is the world's first fully virtualized operating system allowing for the virtualization of the operating system, the storage and the network. In this lab we will introduce each of these virtualization options and more importantly show how they can be combined to create an extremely agile system.Prerequisites
- You have completed the lab Installing Oracle Solaris 11 in Oracle VM VirtualBox.
- If you are not already familiar with ZFS, the Introduction to the Oracle Solaris 11 ZFS File System lab should be completed prior to performing this lab.
The Environment
If you're using the Oracle Solaris 11 VirtualBox VM, the environment is set up as follows:Operating System: Oracle Solaris 11 11/11
Hostname: solaris (default)
IP address: 10.0.2.15
User: oracle (as recommended in the Installation lab)
Password: <user password you have configured at the first boot of Solaris VM>
Root password: <root password you have configured at the first boot of Solaris VM>
Lab Exercises
- Working with Virtual Networking
- Working with Zones
- Basic Zone Administration
- Cloning Zones
- Network Quality of Service
- Delegating Administration
Exercise 1: Working with Virtual Networking
Oracle Solaris 11 introduces a new and powerful network stack architecture which includes:- Networking virtualization with virtual network interface cards (VNICs) and virtual switching (etherstubs)
- Tight integration with zones
- Network resource management - efficient and easy to manage integrated quality of service (QoS) to enforce bandwidth limit on VNICs and traffic flows
- Look at the physical network interfaces on the machine:
oracle@solaris:~$ dladm show-link LINK CLASS MTU STATE BRIDGE OVER net0 phys 1500 up -- --
- Use the physical interface to create a virtual interface. The virtual interface has to be associated with a physical link or virtual switch (etherstub). We'll be associating ours with the physical link:
oracle@solaris:~$ sudo dladm create-vnic -l net0 vnic0 Password:
- The link now shows up as a VNIC:
oracle@solaris:~$ dladm show-link LINK CLASS MTU STATE BRIDGE OVER net0 phys 1500 up -- -- vnic0 vnic 1500 up -- net0
- You can get more details on the virtual interface, including its speed and MAC address, as follows:
oracle@solaris:~$ dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID vnic0 net0 1000 2:8:20:cd:c1:d9 random 0
- Notice it's SPEED matches that of the physical NIC. Say we wanted to limit the bandwitdh on that NIC:
oracle@solaris:~$ sudo dladm set-linkprop -p maxbw=400 vnic0
- View the VNIC again:
oracle@solaris:~$ dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID vnic0 net0 400 2:8:20:cd:c1:d9 random 0
- If you run ifconfig, you'll notice the new VNIC doesn't appear in the output. That's because an IP interface has to be created and address needs to be assigned to it, and the interface brought up. This can be accomplished with ipadm(1M) as follows:
oracle@solaris:~$ sudo ipadm create-ip vnic0 oracle@solaris:~$ sudo ipadm create-addr -T static -a local=10.0.2.20/24 vnic0/v4static1
- Now the VNIC will appear in ifconfig:
oracle@solaris:~$ ifconfig -a4 lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 net0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 3 inet 10.0.2.15 netmask ffffff00 broadcast 10.0.2.255 vnic0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4 inet 10.0.2.20 netmask ffffff00 broadcast 10.0.2.25
- Now you can use the VNIC like any other NIC - assign it to a virtual machine or a zone.
Exercise 2: Working with Zones
Create the Zone
Oracle Solaris Zones are a virtualization technology at the operating system level that allows you to reduce risk by isolating applications into their own secure environment. An Oracle Solaris Zone securely isolates file systems, processes and networks and appears as a separate system on the network. Attacks on the Oracle Solaris Zone are incapable of breaching the Oracle Solaris Zone boundary.In addition to workload isolation, Oracle Solaris Zones also provide the ability to manage system resources dedicated to the workload. Specifically the Oracle Solaris Zone administrator can control the amount of CPU capacity, RAM and virtual memory the Oracle Solaris Zone is allowed to consume. Combined with the network virtualization technology we just examined in Exercise 1, it is also possible to manage the amount of network bandwidth consumed, as we will see in a later exercise.
Graphically, once complete, our machine will be configured to look as follows:
Create a ZFS file system to host the zone (this file system can also be used to house additional zones you decide to create). ZFS is beyond the scope of this lab, but please take the time to complete Introduction to the Oracle Solaris ZFS File System if you're not already familiar with ZFS.
By putting the zone in its own ZFS file system, you will be able to take advantage of ZFS' advanced data management capabilities, such as data compression and the ability to quickly and efficiently clone the zone. We are turning on compression because today's CPUs can compress and uncompress data faster than it takes to write and read uncompressed data to and form the disk. The final component of the zfs create command you see below, rpool/zones, is the name of the dataset, which you would use when applying other ZFS commands to the file system.
oracle@solaris:~$ sudo zfs create -o mountpoint=/zones -o compression=on rpool/zones
- make the virtual interface manually, having it permanently existing regardless of whether the zone is running or not
- use an interface created automatically for the zone when the zone boots.
oracle@solaris:~$ sudo dladm create-vnic -l net0 myzone0
oracle@solaris:~$ dladm show-link LINK CLASS MTU STATE BRIDGE OVER net0 phys 1500 up -- -- vnic0 vnic 1500 up -- net0 myzone0 vnic 1500 up -- net0
- The zonepath is the location on disk where the zone's files reside.
- If the parent directory of the zonepath is a ZFS file system (as is our case), a new ZFS file system is created specifically for the zone. Otherwise, just a directory is created.
- The autoboot setting tells the zone to boot when the system is booted.
- The ip-type of the zone can either be shared or exclusive. The exclusive setting assumes you are dedicating a network interface card for exclusive use by the zone. This brings some network stack management benefits to the zone, but traditionally hasn't been practical as system can have tens or hundreds of zones and generally only a maximum of 4 network interfaces. Fortunately, the new network virtualization features in Solaris 11 allows us to easily overcome this limitation. Exclusive IP is now the recommended ip-type for zones.
- The set physical setting tells the zone which of the system's network interface cards we are assigning to the zone. We are using the new myzone0 VNIC we just created.
- If we were setting the IP type to shared, we would also have to set an IP address as this point. As indicated above, using an IP type of exclusive allows the zone to manage its IP stack directly.
zonecfg The zonecfg utility supports tab completion. Pressing tab at any point will display a list of possible commands, or complete the command if it's uninque. |
oracle@solaris:~$ sudo zonecfg -z myzone myzone: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:myzone> create zonecfg:myzone> set zonepath=/zones/myzone zonecfg:myzone> set autoboot=true zonecfg:myzone> set ip-type=exclusive zonecfg:myzone> add net zonecfg:myzone:net> set physical=myzone0 zonecfg:myzone:net> end zonecfg:myzone> verify zonecfg:myzone> exit
bleonard@solaris:~$ sudo zoneadm -z myzone clone tzone
Identify the two zones configured on our system, the global zone (which is pre-existing on all Oracle Solaris systems), and our new non-global (or local) zone, myzone:
oracle@solaris:~$ zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / ipkg shared - myzone installed /zones/myzone ipkg excl
Note that myzone is installed, but not yet running. We will boot it up very shortly. The brand is ipkg, meaning the zone is running the same operating system as the global zone (Solaris 11). Oracle Solaris 11 also supports branded zones for running other operating systems such as Solaris 10.
The default compression algorithm used by ZFS is lzjb. Now that the zone is installed, check the compression ratio achieved for the rpool/zones dataset:
oracle@solaris:~$ zfs get compressratio rpool/zones NAME PROPERTY VALUE SOURCE rpool/zones compressratio 1.62x -
Prepare to boot the zone. On first boot, the zone walks you through a series of steps to configure itself.
# zoneadm -z myzone boot # zlogin -C myzone
The -C option to zlogin lets us access the zone console, that is, it takes us into the zone and lets us work within the zone. Because no system configuration files are available, the System Configuration Tool starts up, and you have to walk through a number of steps in order to setup time zone, root password and network configuration. The exact procedure of the setup is described in an article on Oracle Technology Network.
Note that (the root password and the password of oracle user is what you configured with System Configuration Tool when connected to the zone with zlogin -C).
Later you can run System Configuration Tools with sysconfig command in order to change the system configuration or to make a profile where system configuration is stored in .xml format.
Exercise 3: Basic Zone Administration
Internet Connectivity
Test if you can ping the outside world from within the zone:
root@myzone:~# ping www.oracle.com www.oracle.com is alive
NOTE: network configuration has been changed in Oracle Solaris 11 comparing to earlier releases. While configuration files such as /etc/resolve.conf and /etc/nsswitch.conf still exist, they are just compiled from the information containing in SMF repository. Thus, you need following commands to check whether IP address, default gateway, nameservers address and name services configured properly:
ipadm show-addr netstat -rn svcprop -p config/nameserver dns/client svcprop name-service/switch
A new utility, nscfg(1), has been provided to import and export name service configuration into and out of the SMF repository and allows legacy files such as /etc/nsswitch.conf and /etc/resolv.conf to regenerate from SMF configuration for backwards compatibility.
You can view the routing table with netstat:
root@myzone:~# netstat -rn Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 10.0.2.2 UG 1 0 10.0.2.0 10.0.2.25 U 4 265 myzone0 127.0.0.1 127.0.0.1 UH 2 24 lo0 Routing Table: IPv6 Destination/Mask Gateway Flags Ref Use If --------------------------- --------------------------- ----- --- ------- ----- ::1 ::1 UH 2 0 lo0
root@myzone:~# ping www.oracle.com www.oracle.com is alive
Remote Access
At this point the zone is only accessible from the global zone using the zlogin utility. To access the zone remotely, a user account needs to be created. For the purpose of this exercise, we will create the user tstark. Please check which actual IP address your myzone interface has, it can be different from 10.0.2.25 which is used below as an example.root@myzone:~# useradd -m -d /tstark -s /usr/bin/bash tstark 80 blocks root@myzone:~# passwd tstark New Password: abc133 Re-enter new Password: abc123 passwd: password successfully changed for tstark
root@myzone:~# passwd -f tstark passwd: password information changed for tstark
Switch back to the other terminal window that's logged into the global zone. For convenience, add myzone to the hosts file as follows:
oracle@solaris:~$ cat /etc/inet/hosts # CDDL HEADER START # # ... # # Internet host table # ::1 solaris solaris.local localhost loghost 127.0.0.1 solaris solaris.local localhost loghost 10.0.2.25 myzone
Then log into myzone as tstark:
oracle@solaris:~$ ssh tstark@myzone The authenticity of host 'myzone (10.0.2.25)' can't be established. RSA key fingerprint is 2c:07:3a:fc:a0:c5:1b:80:de:c8:37:3d:d2:72:56:bf. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'myzone,10.0.2.25' (RSA) to the list of known hosts. Password: abc123 Warning: Your password has expired, please change it now. New Password: 123abc Re-enter new Password: 123abc sshd-kbdint: password successfully changed for tstark Last login: Thu Apr 7 15:15:57 2011 from 10.0.2.15 Oracle Corporation SunOS 5.11 11.0 November 2011 tstark@myzone:~$
Privileged Operations
Our user tstark is pretty toothless. In order to allow him to assume a root role, we'd like to assign it to him. Connect to the zone myzone as a root, and modify tstark account accordingly:usermod -R root tstark
Then login as user tstark again and add user tstark to the sudoers file. To edit the sudoers file you need to use visudo:
VI CheatsheetIf you're unfamiliar with vi, following are a few common keyboard commands to get you though this exercise: k = up j = down w = right or forward one word b = left or back one word l = right 1 char h = left 1 char x = delete 1 char u = undo i = insert dd = delete entire current line [ESC] = get out of edit mode :wq = write and quit :q! = quite ignoring changes (do not write) |
tstark@myzone:~# visudo
Add the following line anywhere in the file. The NOPASSWD: setting is optional, but will make your use of sudo throughout the rest of this lab less painful:
tstark ALL=(ALL) NOPASSWD: ALL
Finally, exit out of the root user:
tstark@myzone:~# exit exit tstark@myzone:~$
Installing Software
We're going to customize this zone a bit so we can show the advantages of cloning a zone in a later exercise. Let's install the apache web servertstark@myzone:~$ sudo pkg install apache-22 Nov 17 04:56:07 solaris sudo: tstark : TTY=console ; PWD=/home/tstark ; USER=root ; COMMAND=/usr/bin/pkg install apache-22 Packages to install: 5 Create boot environment: No Services to restart: 1 DOWNLOAD PKGS FILES XFER (MB) Completed 5/5 907/907 4.7/4.7 PHASE ACTIONS Install Phase 1171/1171 PHASE ITEMS Package State Update Phase 5/5 Image State Update Phase 2/2
tstark@myzone:~$ sudo svcadm enable apache22
Start Firefox in your global zone and test the web server running in the zone:
Exercise 4: Clone the Zone
Now that we've configured the zone the way we like it, we're going to use it as a template for creating additional zones. You'll note how much quicker the creating process is and the zone will be pre-configured just as we want it.The first step is to create another VNIC for the new zone:
oracle@solaris:~$ sudo dladm create-vnic -l net0 myzoneclone0
Next define the zone configuration:
oracle@solaris:~$ sudo zonecfg -z myzoneclone myzoneclone: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:myzoneclone> create create: Using system default template 'SYSdefault' zonecfg:myzoneclone> set zonepath=/zones/myzoneclone zonecfg:myzoneclone> set ip-type=exclusive zonecfg:myzoneclone> add net zonecfg:myzoneclone:net> set physical=myzoneclone0 zonecfg:myzoneclone:net> end zonecfg:myzoneclone> verify zonecfg:myzoneclone> exit
Now we need to provide the system configuration information for the new zone. Use the following. The only changes from the myzone configuration are the hostname and IP address. Let's define IP address for this zone manually, and it is to be 10.0.2.35 in our example. Use another one if 10.0.2.35 does not reflect your network configuration. You'll be prompted to enter appropriate values while sysconfig guides you through configuration.
The configuration can be done prior to installing and booting a zone with configuration profile.
The profile can be created with a command introduced in Oracle Solaris 11:
sysconfig create-profile -o sc_profile.xml
sysconfig If you edited system configuration profile manually and if there's a typo or formatting error in the sysconfig profile, the zone boot process will resort to an interactive mode. |
In order to install a zone using existing system configuration from an .xml file, zoneadm install and zoneadm clone have -c option, followed by a path to a directory, where several directories with different configuration profiles can reside. You can make the appropriate directory structure and copy the configuration profile there:
# mkdir profiles # mkdir profiles/defaultprofile # cp sc_profile.xml profiles/defaultprofile/
Then you can install or clone the zone with the profile created earlier. Use
# zoneadm -z myzoneclone install -c /home/oracle/profilesor
# zoneadm -z myzoneclone clone myzone -c /home/oracle/profiles
The next step is to install the zone, but rather than install, we also have the option to clone an existing zone. First, halt the zone we want to clone:
$ sudo zoneadm -z myzone halt
Then clone myzone:
$ sudo zoneadm -z myzoneclone clone myzone -c /home/oracle/profiles
The system configuration profile mentioned earlier, sc_profile.xml can be made with
sysconfig -o sc_profile.xmlor manually, just by copying and pasting the following text into the file.
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="sysconfig"> <service version="1" type="service" name="system/config-user"> <instance enabled="true" name="default"> <property_group type="application" name="root_account"> <propval type="astring" name="login" value="root"/> <propval type="astring" name="password" value="$5$IP.0cabs$/3rNMAzh0lP/v/hLzp5iyClfTaeoln.VkgT0raL1bT9"/> <propval type="astring" name="type" value="normal"/> </property_group> </instance> </service> <service version="1" type="service" name="system/timezone"> <instance enabled="true" name="default"> <property_group type="application" name="timezone"> <propval type="astring" name="localtime" value="US/Pacific"/> </property_group> </instance> </service> <service version="1" type="service" name="system/environment"> <instance enabled="true" name="init"> <property_group type="application" name="environment"> <propval type="astring" name="LANG" value="C"/> </property_group> </instance> </service> <service version="1" type="service" name="system/identity"> <instance enabled="true" name="node"> <property_group type="application" name="config"> <propval type="astring" name="nodename" value="myzoneclone"/> </property_group> </instance> </service> <service version="1" type="service" name="system/keymap"> <instance enabled="true" name="default"> <property_group type="system" name="keymap"> <propval type="astring" name="layout" value="US-English"/> </property_group> </instance> </service> <service version="1" type="service" name="system/console-login"> <instance enabled="true" name="default"> <property_group type="application" name="ttymon"> <propval type="astring" name="terminal_type" value="sun-color"/> </property_group> </instance> </service> <service version="1" type="service" name="network/physical"> <instance enabled="true" name="default"> <property_group type="application" name="netcfg"> <propval type="astring" name="active_ncp" value="DefaultFixed"/> </property_group> </instance> </service> <service version="1" type="service" name="network/install"> <instance enabled="true" name="default"> <property_group type="application" name="install_ipv4_interface"> <propval type="astring" name="address_type" value="static"/> <propval type="net_address_v4" name="static_address" value="10.0.2.35/24"/> <propval type="astring" name="name" value="myzoneclone0/v4"/> </property_group> <property_group type="application" name="install_ipv6_interface"> <propval type="astring" name="stateful" value="yes"/> <propval type="astring" name="stateless" value="yes"/> <propval type="astring" name="address_type" value="addrconf"/> <propval type="astring" name="name" value="myzoneclone0/v6"/> </property_group> </instance> </service> <service version="1" type="service" name="system/name-service/switch"> <property_group type="application" name="config"> <propval type="astring" name="default" value="files"/> <propval type="astring" name="printer" value="user files"/> </property_group> <instance enabled="true" name="default"/> </service> <service version="1" type="service" name="system/name-service/cache"> <instance enabled="true" name="default"/> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"/> </service> </service_bundle>
# zoneadm -z myzoneclone boot # zlogin -C myzoneclone [Connected to zone 'myzoneclone' console] Hostname: unknown Hostname: myzoneclone myzoneclone console login: Nov 18 03:37:55 myzoneclone sendmail[9688]: My unqualified host name (myzoneclone) unknown; sleeping for retry Nov 18 03:38:55 myzoneclone sendmail[9688]: unable to qualify my own domain name (myzoneclone) -- using short name myzoneclone console login: tstark Password: 123abc Oracle Corporation SunOS 5.11 11.0 November 2011 tstark@myzoneclone:~$ ifconfig -a4 lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 myzoneclone0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.0.2.35 netmask ffffff00 broadcast 10.0.2.255
Return to Firefox and test the web server.
It's important to have 10.0.2.35 to be routable from your global zone. If you have another IP address configured for the myzoneclone zone, use it instead. It has to be appropriate for your network configuration.
We'll just use the IP address rather than taking the time to set the hostname:
User tstark is also available in the clone. It even detects a "last login", which was actually from myzone:
oracle@solaris:~$ ssh tstark@10.0.2.35 The authenticity of host '10.0.2.35 (10.0.2.35)' can't be established. RSA key fingerprint is 8e:42:ea:ca:9c:dd:89:ba:80:a0:d1:60:74:79:4b:a2. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '10.0.2.35' (RSA) to the list of known hosts. Password: 123abc Last login: Thu Apr 7 15:15:57 2011 from 10.0.2.15 Oracle Corporation SunOS 5.11 11.0 November 2011 tstark@myzoneclone:~$
Exercise 5: Network QoS
Thus far we've seen the powerful combination of operating system virtualization and storage virtualization in that we were able to get a clone of an operating environment up and running in just a couple of minutes. In this exercise we are going to look at the additional value network virtualization brings to our zones.From the global zone, ping myzoneclone with 10K of data as follows:
oracle@solaris:~$ ping -s 10.0.2.35 10000 PING 10.0.2.35: 10000 data bytes 10008 bytes from 10.0.2.35: icmp_seq=0. time=1.513 ms 10008 bytes from 10.0.2.35: icmp_seq=1. time=0.973 ms 10008 bytes from 10.0.2.35: icmp_seq=2. time=1.079 ms 10008 bytes from 10.0.2.35: icmp_seq=3. time=1.718 ms ...
Let's assume this particular zone was something like a low priority network tape backup service. However, when it kicked in to perform it's daily backup, it bogged down the rest of the network with it's backup data traffic, creating a quality of service issue. By utilizing resource controls on the virtual network interface, we can easily limit the amount of bandwidth that this particular interface is allowed to consume, therefore freeing up bandwidth for other consumers on the network.
View the VNICs:
oracle@solaris:~$ dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID vnic0 net0 400 2:8:20:7b:6e:37 random 0 myzone0 net0 1000 2:8:20:bf:e7:8c random 0 myzoneclone0 net0 1000 2:8:20:e:35:f1 random 0
oracle@solaris:~$ sudo dladm set-linkprop -p maxbw=8 myzoneclone0 Password: oracle
You should immediately see your ping response times jump by a factor of about 10:
10008 bytes from 10.0.2.35: icmp_seq=24. time=10.147 ms 10008 bytes from 10.0.2.35: icmp_seq=25. time=10.279 ms 10008 bytes from 10.0.2.35: icmp_seq=26. time=10.295 ms
The other important thing to note is that the change was instant and dynamic. There is no need for anything to be restarted in order to make changes to our network traffic.
In this exercise we set a bandwidth setting to the entire VNIC. However, it is also possible to configure bandwidth based on IP address, port, protocol or MAC address, so even different network flows over the same NIC can be adjusted. For more information on this topic, see Oracle Solaris 11 Networking Virtualization Technology page .
Exercise 6: Delegate Administration
A new feature of Solaris 11 is the ability to delegate the administration of a zone to another user in the global zone. For this exercise, we first to to create a new user in the global zone, ppotts:oracle@solaris:~$ sudo useradd -m -d /export/home/ppotts -s /usr/bin/bash ppotts 80 blocks
oracle@solaris:~$ sudo passwd ppotts New Password: abc123 Re-enter new Password: abc123 passwd: password successfully changed for ppotts
oracle@solaris:~$ su - ppotts Password: ppotts@solaris:~$ pfexec zoneadm -z myzoneclone halt zoneadm: zone 'myzoneclone': only a privileged user may halt a zone.
pfexec pfexec is used to run commands against a user's profile. In the next step we are going to assign Pepper the authorizations she needs to administer the zone. |
Now let's give Pepper the privilege to administer myzoneclone:
ppotts@solaris:~$ exit exit oracle@solaris:~$ sudo zonecfg -z myzoneclone zonecfg:myzoneclone> add admin zonecfg:myzoneclone:admin> set user=ppotts zonecfg:myzoneclone:admin> set auths=login,manage,clonefrom zonecfg:myzoneclone:admin> end zonecfg:myzoneclone> verify zonecfg:myzoneclone> exit
ppotts@solaris:~$ pfexec zoneadm -z myzoneclone halt ppotts@solaris:~$ zoneadm list -cv ID NAME STATUS PATH BRAND IP 0 global running / ipkg shared - myzone installed /zones/myzone ipkg excl - myzoneclone installed /zones/myzoneclone ipkg excl
What happens if Pepper tries to manage a different zone?
Summary
Congratulations, you've just employed 3 types of virtualization to create a more agile data center. By combining network virtualization with ZFS and zones you can establish environments which can be quickly replicated and controlled.false ,,,,,,,,,,,,,,,,
Comentarios
Publicar un comentario