Ping@ubuntu1:$ cat /etc/os-releaseNAME='Ubuntu'VERSION='14.04.1 LTS, Trusty Tahr'ID=ubuntuIDLIKE=debianPRETTYNAME='Ubuntu 14.04.1 LTS'VERSIONID='14.04'HOMEURL='cat /etc/lsb-releaseDISTRIBID=UbuntuDISTRIBRELEASE=14.04DISTRIBCODENAME=trustyDISTRIBDESCRIPTION='Ubuntu 14.04.2 LTS'ping@trinity:$ cat /etc/issueUbuntu 14.04.2 LTS n lping@trinity:$ cat /etc/issue.netUbuntu 14.04.2 LTSping@ubuntu1:$ cat /proc/versionLinux version 3.13.0-32-generic (buildd@kissel)(gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) )#57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014. Ping@trinity:$ lscpuArchitecture: x8664CPU op-mode(s): 32-bit, 64-bitByte Order: Little EndianCPU(s): 32 #.
Ping@trinity:$ grep -m1 'model name' /proc/cpuinfomodel name: Intel(R) Xeon(R) CPU E5-4627 v2 @ 3.30GHz #. Ping@trinity: sudo dmidecode -t 17# dmidecode 2.12SMBIOS 2.8 present.Handle 0x1100, DMI type 17, 40 bytesMemory DeviceArray Handle: 0x1000Error Information Handle: Not ProvidedTotal Width: 72 bitsData Width: 64 bitsSize: 32 GB #. Ping@trinity:$ less /boot/config-3.19.0-25-generic grep -i iommuCONFIGGARTIOMMU=yCONFIGCALGARYIOMMU=yCONFIGCALGARYIOMMUENABLEDBYDEFAULT=yCONFIGIOMMUHELPER=yCONFIGVFIOIOMMUTYPE1=mCONFIGIOMMUAPI=yCONFIGIOMMUSUPPORT=y #. Ping@trinity:$ modinfo kvmfilename: /lib/modules/3.19.0-25-generic/kernel/arch/x86/kvm/kvm.kolicense: GPLauthor: Qumranetsrcversion: F58A0F8858A02EFA0549DE5depends:intree: Yvermagic: 3.19.0-25-generic SMP modunload modversionssigner: Magrathea: Glacier signing keysigkey: 6A:AA:11:D1:8C:2D:3A:40:B1:B4:DB:E5:BF:8A:D6:56:DD:F5:18:38sighashalgo: sha512parm: allowunsafeassignedinterrupts:Enable device assignment on platforms without interrupt remapping support. (bool)parm: ignoremsrs:boolparm: mintimerperiodus:uintparm: tsctoleranceppm:uint.HW acceleration OK. Handle 0x0000, DMI type 0, 24 bytesBIOS InformationVendor: HPVersion: I32 #. Ping@Compute24:$ dmesg -T grep -iE 'iommu dmar'Thu Dec 17 09: Command line: BOOTIMAGE=/vmlinuz-3.16.0-30-genericroot=/dev/mapper/Compute24-vg-root ro inteliommu=oncrashkernel=384M-2G:64M,2G-16G:128M,16G-:256MThu Dec 17 09: ACPI: DMAR 0x00000000BDDAB840 000718 (v01 HPProLiant 00000001 xffffffd2?
Currently the ixgbe coming with ubuntu does not work on VMX. The main issue islack of multicast support on ingress - packet received on a VF will bediscarded siliently and won’t be delivered into the guest VM, an example of theimmediate effect of this is that OSPF (and most of today’s IGP) neighborshipwon’t come up.
Therefore building VMX based on SR-IOV requires to re-compilethe ixgbe kernel driver from source code, which is provided by Juniper to fixthe multicast support. The code is available in the installation package. Atthe time of the writing of this document there is problem to compile ixgbe fromsource code under any kernels other than 3.13.0-32-generic. That’s why thekernel needs to be changed in this setup. Sudo apt-get install bridge-utils qemu-kvm libvirt-bin python numactl python-netifaces vnc4server libyaml-dev python-yaml libparted0-dev libpciaccess-dev libnuma-dev libyajl-dev libxml2-dev libglib2.0-dev libnl-dev libnl-dev python-pip python-dev libxml2-dev libxslt-devReading package lists. DoneBuilding dependency treeReading state information. Cd libvirt-1.2.8sudo./configure -prefix=/usr/local -with-numactlchecking for a BSD-compatible install.
/usr/bin/install -cchecking whether build environment is sane. Yeschecking for a thread-safe mkdir -p. /bin/mkdir -pchecking for gawk.
. Virtualized Juniper MX platform with complete control, forwarding. VirtIO and SR-IOV capable for high throughput. Can be hosted in VM or bare-metal Virtual Forwarding Plane (VFP). Juniper Networks Corporate PowerPoint Template Author. Juniper Networks vMX on Snabb NFV. This is my first attempt getting Juniper Networks vMX virtual router running on top of Snabb Switch. This is probably one of the first few attempts to run non-Linux based VM’s on top of Snabb NFV and more work is need to get it fully functional.
Gawkchecking whether make sets $(MAKE). Yes.sudo service libvirt-bin stoplibvirt-bin stop/waitingsudo make uninstallMaking uninstall in.make1: Entering directory `/tmp/libvirt-1.2.8'make1: Leaving directory `/tmp/libvirt-1.2.8'./bin/rm rf /usr/local/lib/libvirt.in/rm:cannot remove ‘/usr/local/lib/libvirt.’: No such file or directory.install new version. Sudo./configure -prefix=/usr -localstatedir=/ -with-numactlchecking for a BSD-compatible install. /usr/bin/install -cchecking whether build environment is sane. Yeschecking for a thread-safe mkdir -p.
/bin/mkdir -p.sudo makemake all-recursivemake1: Entering directory `/tmp/libvirt-1.2.8'Making all in.make2: Entering directory `/tmp/libvirt-1.2.8'make2: Leaving directory `/tmp/libvirt-1.2.8'Making all in gnulib/libmake2: Entering directory `/tmp/libvirt-1.2.8/gnulib/lib'GEN alloca.hGEN cdefs.hGEN warn-on-use.hGEN arg-nonnull.hGEN arpa/inet.h.sudo make installMaking install in.make1: Entering directory `/tmp/libvirt-1.2.8'make2: Entering directory `/tmp/libvirt-1.2.8'.start new version. Ping@trinity:$ libvirtd -versionlibvirtd (libvirt) 1.2.8ping@trinity:$ service libvirt-bin statuslibvirt-bin start/running, process 1559ping@trinity:$ which libvirtd/usr/sbin/libvirtdping@trinity:$ /usr/sbin/libvirtd -version/usr/sbin/libvirtd (libvirt) 1.2.8ping@trinity:$ virsh -version1.2.8ping@trinity:/images/vmx20151102.0$ sudo virsh -c qemu:///system versionCompiled against library: libvirt 1.2.8Using library: libvirt 1.2.8Using API: QEMU 1.2.8Running hypervisor: QEMU 2.0.0. Pings@svl-jtac-tool02:/volume/build/junos/15.1F/daily/20151102.0/ship$ls -l grep install64 grep vmx1 builder 748510583 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz1 builder 33 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz.md51 builder 41 jinstall64-vmx-15.1F-20151102.0-domestic-signed.tgz.sha11 builder jinstall64-vmx-15.1F-20151102.0-domestic.img #. Ping@trinity:/virtualization/images/vmx20151102.0/config$ cat vmx.conf ############################################################## # # vmx.conf # Config file for vmx on the hypervisor. # Uses YAML syntax. # Leave a space after ':' to specify the parameter value.
# ##############################################################-#Configuration on the host side - management interface, VM images etc.' HOST' config section.ID of the vmx instance. This string will be encoded into the final VM name:vcp- or vfp-.current management interface of the server. The installation script will 'move'the IP/MAC property from this port to an external bridge named 'br-ext'.vRE/vFPC/Harddisk VM images location.external bridge configuration section: a bridge utility, named br-ext,will be created, for managment connection from/to the external networks.vRE configuration section: this template uses 1 vCPU, 1G mem, console port8601 to start vRE guest VM. The VM mgmt interface’s 'peer interface'from the host - vcpext-vmx1 ('attached' to fxp0 port from inside theguest VM), will be configured with the specified MAC address.
The correspondingfxp0 interface from inside of vRE VM will inherit same MAC from it.vFPC configuration section: this template uses 3 vCPU, 6G mem, console port8602 to build vFPC guest VM. The VM mgmt interface’s 'peer interface' from the host - vfpext-vmx1 ('attached' toext port from side the guest VM) will be configured with the specified MACaddress.
The ext interface from inside the vFPC guest VM will inherit sameMAC from it.as a KVM implementation, VMX currently supports two type of network IOvirtualization: VT-d + SRIOV, or VIRTIO. This config knob device-typewill determine which IO virtualization technology will be used to build VMX.This template uses 'virtio' IO virtualization.VMX router interface configuration section: This is where the routerge-0/0/z properties can be configured. Depending on device-type value theavailable configurable properties will be different. Since this template uses'virtio' virtualization, only 'mac-address' is configurable. More details willbe covered in later sections of this doc.locally administered MAC address.last 4 digits of IP (e.g. X.y.4.17) or MAC of management interface.VMX instance number, first VMX instance uses 01, second uses 02, etc.01 for control plane interface ( fxp0 for VCP, ext for VFP) of a VM,02 for forwarding plane interface (ge-0/0/z) of a VM.assign a unique number for each type of interfaceTable 3.
An example of MAC address allocation with this ruleVMX instance1vRE fxp002:04:17:01:01:01vFPC ext02:04:17:01:01:02ge-0/0/002:04:17:01:02:01ge-0/0/102:04:17:01:02:02VMX instance2vRE fxp002:04:17:02:01:01vFPC ext02:04:17:02:01:02ge-0/0/002:04:17:02:02:01ge-0/0/102:04:17:02:02:02. Ping@trinity:/virtualization/images/vmx20151102.0/config$ cat vmx.conf ############################################################## # # vmx.conf # Config file for vmx on the hypervisor. # Uses YAML syntax.
# Leave a space after ':' to specify the parameter value. # ##############################################################-#Configuration on the host side - management interface, VM images etc. Key parameters in this conf file:.my server’s mgmt interface name is em1.vRE and vFPC images location will be just the images folder from the untar.edinstallation package. These images can then be shared by all VMX instances.virtual harddisk image vmxhdd.img will be in a seperate folder createdspecifically for current VMX instance.use console port 88x6 for vRE guest VM and 88x7 for vFPC guest VM, where x =instance number. Any other number which is not yet in use is fine.MAC addresses are allocated following.for SR-IOV virtualization, these link properties need to be configured.for virtio virtualization, only MAC address can be defined in this config file.MTU can to be configured in a seperate junosdev.conf config file.virtio technology generates seperated virtual NICs that are used to buildVMX; all L1 properties remains in the physical NIC.
Therefore no need tospecify L1 properties like physical NIC, VF, link speed, etc for VIRTIO setup.To communicate with external networks, virtio virtual NIC can be 'bound' toa physical NIC port, or another virtual NIC, using any existing technologieslike linux bridge, OVS, etc. Ping@trinity:/virtualization/images/vmx20151102.0/config$ cat vmx.conf.virtio.1 ############################################################## # # vmx.conf # Config file for vmx on the hypervisor. # Uses YAML syntax. # Leave a space after ':' to specify the parameter value. # ##############################################################-#Configuration on the host side - management interface, VM images etc.
![Juniper Vmx Sr Iov Juniper Vmx Sr Iov](/uploads/1/2/5/4/125423168/904933548.png)
Ping@trinity:/virtualization/images/vmx20151102.0$ sudo./vmx.sh -cleanup -cfg config/vmx.conf.virtio.1 Welcome to VMX Date.11/30/15 21:14:08 VMX Identifier.vmx1 Config file. Ping@trinity:/virtualization/images/vmx20151102.0$ cat config/vmx.conf.virtio.1 ############################################################## # # vmx.conf # Config file for vmx on the hypervisor. # Uses YAML syntax. # Leave a space after ':' to specify the parameter value. # ##############################################################-#Configuration on the host side - management interface, VM images etc. Ping@trinity:/virtualization/images/vmx20151102.0$ cat config/vmx.conf.virtio.2 ############################################################## # # vmx.conf # Config file for vmx on the hypervisor.
# Uses YAML syntax. # Leave a space after ':' to specify the parameter value.
# ##############################################################-#Configuration on the host side - management interface, VM images etc. Ping@trinity:/virtualization/images/vmx20151102.0$ cat config/vmx-junosdev.conf.1 ############################################################## # # vmx-junos-dev.conf # - Config file for junos device bindings. # - Uses YAML syntax. # - Leave a space after ':' to specify the parameter value.
On the other hand, the script does nothing but to prepare the work environmentfor the VMX and run some commands to bring up the two VM images. Therefore,having the script does not stop us from doing all of the installation workmanually. One advantage of doing so is that if for any reason the script bailsout (e.g sometime the server does not meet all of the presumptions or criteriasthe script requires in order to run smoothly), we can still fine tune thesystem manually and proceed the installation task. But in order to do that, weneed to know what exactly the script does so we can at least use the step as agood and tested-and-working reference when something went wrong in the middle. Ping@matrix:$ sudo mount -t hugetlbfs hugetlbfs /HugePagevPFEping@matrix:$ sudo virsh start AVPN-vfperror: Failed to start domain AVPN-vfperror: internal error: hugetlbfs filesystem is not mounted or disabled by administrator configping@matrix:$ ls /HugePagevPFE/ping@matrix:$ sudo service libvirt-bin restartlibvirt-bin stop/waitinglibvirt-bin start/running, process 46220ping@matrix:$ ls /HugePagevPFE/libvirtping@matrix:$ sudo virsh start AVPN-vfpDomain AVPN-vfp started.prepare for pci-stub (SR-IOV only).
Sudo -i#p3p1 vf0 echo 0000:23:10.0 /sys/bus/pci/devices/0000:23:10.0/driver/unbindecho 0000:23:10.0 /sys/bus/pci/drivers/pci-stub/bind#p3p1 vf1 echo 0000:23:10.2 /sys/bus/pci/devices/0000:23:10.2/driver/unbindecho 0000:23:10.2 /sys/bus/pci/drivers/pci-stub/bind#p2p1 vf0 echo 0000:06:10.0 /sys/bus/pci/devices/0000:06:10.0/driver/unbindecho 0000:06:10.0 /sys/bus/pci/drivers/pci-stub/bind#p2p1 vf1 echo 0000:06:10.2 /sys/bus/pci/devices/0000:06:10.2/driver/unbindecho 0000:06:10.2 /sys/bus/pci/drivers/pci-stub/bindexit.config VN & bridges. #vmx1: export XMLFOLDER= '/virtualization/vmx1 'sudo virsh net-define $XMLFOLDER/br-ext-generated.xmlsudo ifconfig em1 0; sudo virsh net-start br-ext; sudo brctl addif br-ext em1; sudo route add default gw 10.85.4.1sudo ifconfig br-ext hw ether 38:ea:a7:37:7c:54;sudo virsh net-define $XMLFOLDER/br-int-generated.xmlsudo virsh net-start br-int-vmx1#vmx2: export XMLFOLDER= '/virtualization/vmx2 'sudo virsh net-define $XMLFOLDER/br-int-generated.xmlsudo virsh net-start br-int-vmx2.define and start VMs using the generated xml file. [email protected]:$ sudo netstat -nap grep qemusudo password for ping:tcp 0 0 127.0.0.2:8896 0.0.0.0:. LISTEN 33284/qemu-system-x #. Right after a fresh VMX installation, the guest VCP/VFP VM will come up with noconfigs. So the initial configuration (mgmt IP, GW, login, etc) needs to bedone via a 'console' connection. Previously we demonstrated how to login tothe vRE and vPFE console via telnet, which is also a convenient method tocollect the guest VM booting messages in the case that IP-based managementsession is not available.
Another way to acquire 'console' of a guest VM is viathe built-in VNC support provided by KVM. Root@ubuntu1:# modinfo ixgbevffilename: /lib/modules/3.13.0-32-generic/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.koversion: 2.11.3-klicense: GPLdescription: Intel(R) 82599 Virtual Function Driverauthor: Intel Corporation, srcversion: AE2D8A11E943Dalias: pci:v0001515sv.sd.bc.sc.i.alias: pci:v00010EDsv.sd.bc.sc.i.depends:intree: Yvermagic: 3.13.0-32-generic SMP modunload modversionssigner: Magrathea: Glacier signing keysigkey: 5E:3C:0F:9C:A6:E3:65:43:53:5F:A2:BB:5B:70:9E:84:F1:6D:A7:C7sighashalgo: sha512parm: debug:Debug level (0=none.,16=all) (int).