Sunyit's Project BITS Documentations

The following is a collection of documents that I created solely for myself and colleagues in order to meet standards for implementing a Hadoop cloud service. That said there is a lot of information that is specific for the systems used and customized to only work for those who were apart the project. The objective for Project "Bits" can be found here in this link. All ip addresses have been marked with x's and urls generalized in order to protect the SunyIT network system. I continue to study the systems used here and release the documents in hope that others might take up the project and implement it at his or her's University/College.


Back-Bone of Bits Project
  • This is the server BitsGW which features a vpn connection across multiple colleges. Creating VM’s of BitsHP (hadoop machines) to have a scalable new projects. Also providing a LDAP connection service.
  • BitsGW has the following user: afassett, admin
  • BitsHP: Pxe server for machines behind it for Hadoop nodes. Which are booted dynamically without drives over the network.
  • Nodes: clients to the master, using dsl version of linux to size down on amount of memory the os uses to saving space.
Creating the BitsGW  (Building the Vpn Service)e
  1. Install vpn service
    • root@bitsGW:apt-get install openvpn
  2. using VPN connection using files from colleges.(there should be a key and conf files)
    • Grabbing the files from USB
      • root@bitsGW:mkdir -p /mnt/usb
      • root@bitsGW:grep SCSI /var/log/messages|less (find what the usb is called)
      • root@bitsGW:mount -t vfat -o rw, users /dev/sdb1 /mnt/usb
  3. Input files
    • root@bitsGW:mv client-sunyit.conf /etc/openvpn
    • root@bitsGW:mv lair /etc/openvpn
  4. Start Vpn with configurations
    • root@bitsGW:openvpn [client config file]
    • to simplify input this → /etc/openvpn/client-sunyit.conf
Resources:HOWTO: OpenVPN

Created a directory for “log” and inside it is logs for any errors on the file "backbone.log". I also made sure that the key file was chmod 600 or 700 either does not affect the process. Another problem encountered was using the permission script on openvpn you may need to use the following.

openvpn --script-securitory 2 --config client-sunyit.conf
Building the Ldap Service

Installing Ldap server using Version of Ldap 2.4.33

Commands
  1. Getting the file... wget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-.4.33.tgzwget ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.33.tgz
  2. opening the file... gzip -dc openldap-2.4.33.tgz |tar x
  3. More Documents to read on SSL connections... http://www.openldap.org/doc/admin24/install.html
  4. Run configuration... Results: configure: error: BDB/HDB: BerkeleyDB not available

Download: wget download.oracle.com/otn/berkeley-db/db-5.3.21.tar.gz This will not work!!!
Must have a account and download it manually...

Berkely Installation

Used scp, downloaded the file from my windows machien to the linux server. Info on scp for help...

 Other Research
Find a Google drive command line that looked interesting...may work but not going to go into it right now will just send it through a usb..


Network Research - Of Day 1
This is a record of all the network problems faced & how to debug them.

 Resources
Network Diagnose

Testing on Linux
I tried with the following.../etc/network/interfaces, two files of interests 
/syslinux/syslinux.cfg &  
/syslinux.cfg.
  • auto lo
  • iface lo inet loopback
  • allow-hotplug eth0
  • iface eth0 inet static
  • address x.x.x.x
  • netmask x.x.x.x
  • network x.x.x.x
  • broadcast x.x.x.x
  • gateway x.x.x.x
  • #dns-* options are implemented by the resolvconf package if installed
  • #dns-nameservers x.x.x.x
  • #dns-search cs.sunyit.edu
Other commands tried...
  • /etc/resolv.conf
  • search cs.sunyit.edu
  • nameserver x.x.x.x
  • >route
  • >arp -a 
  • >sudo netstat -natp | grep sshd ← see if the service is running and what ip/ports listening on.
RESULT
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1180/sshd
tcpd 0 0 :::22 :::* LISTEN 11180sshd
Conclusion: all correct.

With a Windows machine I tried with a static IP and a dynamic IP, I wanted to see if this was my linux setup or just how I was doing it the process. In both case nothing worked and did not have any other settings.

Important commands
  • inside file /etc/rc.conf
  • /etc/init.d/networking stop
  • /etc/init.d/networking start
  • ifconfig_de2="inet x.x.x.x netmask x.x.x.x"

Network Resources of Day 2

With nick help learning new commands!
  • dhclient eth0
  • ifup eth0

This script lets me know whats on the network!
for i in {1..10}; do ping -c1 10.107.1.$i; done

Resource

http://serverfault.com/questions/73360/assigning-sequential-dynamic-ip-addresses-in-dhcpd

 Building with real world (PXE Server)
File of importance - /boot/syslinux/syslinux.cfg
apt-get install syslinux

Linux Tutorial - Linux DHCP Server Configuration


Diskless Server
DisklessUbuntuHowto - Community Ubuntu Documentation
PXELINUX - Syslinux Wiki
apt - How to customize live Ubuntu CD? - Ask Ubuntu - Here is a way to modify the iso we get to clients..

Notes on Hadoop
Java SDK Issues
Problems with Source.list
Debian User Forums • View topic - Can't install anything with apt

Other Cloud Research

Popular posts from this blog

UI-Bootstrap Collapsible Sticky Footer

    I could have done this a lot of ways and ultimately I made it work with javascript modifying the class names. I also had to create multiple class names which I didn't like to much, I might come back and improve this but for now this is was I got. Also I call bootstrap+angularjs = ui-bootstrap, not sure if that's common sense but thought I should include it somewhere in my notes. Working Example: http://plnkr.co/edit/2N9ENlbw3bSvg1W9n0ch?p=preview This little css was key to prevent any kind of animation on the collapse, which would result in a scrollbar appearing on the body. A scrollbar can appear when the content's too long but not on the sticky foot duh! .no-animate { transition-duration: 0.0125s; } Research Angular-UI Bootstrap Collapse without animation - Stack Overflow http://jsfiddle.net/paulalexandru/Z2Lu5/  - Was looking at this to see if I could reuse it but I ended but just doing my own thing. Another way which I've seen Using a sticky
Read more

Installing Windows on Acer chromebook 15 cb3-532

Much of my notes are just small problems I faced (what's new!) haha. Either way this was a really fun project to do for a friend and I got excited once I research all the information. You can find instruction videos but what helped me the most was the coolstar website. Sadly they don't have parameters in the url so I can link the machine I used in this article. -  https://coolstar.org/chromebook/windows-install.html Scared of messing with the bios? Don't be!    So you need to first remove the screw to put a legacy bios on the machine. Its easy to backup the chrome os one and with the same steps to change it. You can go back in and revert to the chrome version bios. Its that last part that had me worried and its very easy to do.  Problems getting into Developer Mode?   Follow the instructions to the letter, during the step to disable the OS verification, press Ctrl-D to boot into os. If you hear two beeps I would go back to set one. I got in a loop where it s
Read more

Aspetcore: NLog with Postgresql

This Article is base ASP.Net Core 2 and the latest version of Nlog. 1. Install NLog and the Postgresql Package Using NuGet to install the package into your ASP.NET Core project: PM > Install-Package NLog.Web.AspNetCore PM > Install-Package Npgsql 3. Register NLog Add in your Startup.cs  public void Configure ( IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory ) { ... env.ConfigureNLog( "nlog.config" ); ... // make sure Chinese chars don't fk up Encoding.RegisterProvider(CodePagesEncodingProvider.Instance); //add NLog to ASP.NET Core loggerFactory.AddNLog(); //add NLog.Web app.AddNLogWeb(); } 4. Add to Your Controller Add a logger object to your Controller. private readonly ILogger<YourController> _logger; Modify the constructor to use it public YourController ( ILogger<YourController> logger = null ) { if ( null != logger) { _logger = logger;
Read more