EC2 Vimfrastructure

 Posted on October 22, 2020  |  Vlas Sokolov

Jupiter notebooks from an EC2 server!

I use AWS EC2 instances quite a bit - the headless, Linux AMI kind. The kind where the first ssh login lands you in a bare and minimalistic system and gives a distinct metallic aftertaste. The kind that I - grown soft with friendly UIs and cozy zsh themes - can’t use any longer without daydreaming of a keyboard flying out the window. So, mostly for self-use, here’s how I’ve set up the recent EC2 server.

Some common requirements first:

  1. Python, the pyenv variety - with scientific / numerical / however-the-youngsters-call-it-nowadays stack installed on top of it.
  2. Vim + ale + pylint + python-language-server as a yes-modern-none-of-your-beeswax IDE.
  3. Ipython (because Vim is a perfect IDE and I never have to try things out in an ipython session on the side).
  4. Jupyter notebook server, allowing for remote client connection.

Step 1: first login

Log into the AWS EC2 machine with the .pem key provided.

Then, establish the base habitat:

  1. Set up base habitat: sudo yum install git vim htop zsh xeyes
  2. Get oh-my-zsh: sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
  3. Change bash –> zsh: sudo chsh -s $(which zsh) $(whoami)
  4. Tinker with ~/.zshrc until happy
  5. (Optional): arrange for the return of the Sage Cow

Step 2: system packages

We need to get them pesky system libraries installed. Takes a bit of tinkering, details vary on the individual Linux AMI details:

  1. Install this: yum groupinstall "Development tools"
  2. More packages to avoid the unfortunate pandas catastrophe of October 10th: sudo yum install zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel xz xz-devel libffi-devel
  3. Enable an occasional X11-forwarding (self-note: ahem, on a headless system, why?!): sudo yum install xauth
  4. pyenv and OpenCV need this: sudo yum install openssl-devel readline-devel zlib-devel sqlite-devel mesa-libGL

Step 3: pyenv

sudo yum install pyenv
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.zshrc
pyenv install 3.8.5
echo 'pyenv global 3.8.5' >> ~/.zshrc

Step π: vim + ale + pylint = IDE

Add the following as .vimrc:

syntax on
colorscheme desert
set number
set hlsearch
set ruler

" indentations:
filetype plugin indent on
" show existing tab with 4 spaces width
set tabstop=4
" when indenting with '>', use 4 spaces width
set shiftwidth=4
" On pressing tab, insert 4 spaces
set expandtab

" stop yelling at me VIM!
cnoreabbrev W w

" ukrainian vim
"nmap Ж :
"nmap Жй :q
"nmap Жц :w

" highlight trailing whitespaces
highlight ExtraWhitespace ctermbg=magenta guibg=magenta
match ExtraWhitespace /\s\+$/

Then run the following to install pylint / python language server (do I really need that [all] tag?) and set up ale:

pip install pylint 'python-language-server[all]'
mkdir -p ~/.vim/pack/git-plugins/start
git clone --depth 1 https://github.com/dense-analysis/ale.git ~/.vim/pack/git-plugins/start/ale

Add the following to your .pylintrc file to catch those pesky NumPy / PyTorch false-positive errors in pylint:

[TYPECHECK]

# List of members which are set dynamically and missed by pylint inference
# system, and so shouldn't trigger E1101 when accessed. Python regular
# expressions are accepted.
generated-members=numpy.*, torch.*

Step 4: let the spice flow

Press the big red pip install matplotlib numpy scipy ipython jupyter opencv-python and pray I remembered to list all the libs dependencies above.

Step 5: run the notebook server

I used screen to keep it running even if the ssh connection breaks down:

screen -S jupyter jupyter notebook --no-browser --port=8888

And then Ctrl + A followed by D to detach from screen

Step 6: connect from remote

Establish an ssh connection with port forwarding (remember open the proper ports in AWS security groups):

ssh -i ~/.ssh/ec2-access-key.pem -L 8000:localhost:8888 ec2-user@xx.xx.xxx.xx

Then connect to the running notebook server in the browser!

http://localhost:8000/notebooks/

Step I’ve-lost-count: start actually working!

See, it was, ahem, very easy! So next time, just use Ubuntu image just make a Docker container already be stubborn and keep installing out-of-scope stuff on a headless machine for no apparent reason.