Jupiter notebooks from an EC2 server!
I use AWS EC2 instances quite a bit - the headless, Linux AMI kind. The kind where the first ssh login lands you in a bare and minimalistic system and gives a distinct metallic aftertaste. The kind that I - grown soft with friendly UIs and cozy zsh themes - can’t use any longer without daydreaming of a keyboard flying out the window. So, mostly for self-use, here’s how I’ve set up the recent EC2 server.
Some common requirements first:
- Python, the pyenv variety - with scientific / numerical / however-the-youngsters-call-it-nowadays stack installed on top of it.
- Vim + ale + pylint + python-language-server as a yes-modern-none-of-your-beeswax IDE.
- Ipython (because Vim is a perfect IDE and I never have to try things out in an ipython session on the side).
- Jupyter notebook server, allowing for remote client connection.
Step 1: first login
Log into the AWS EC2 machine with the
.pem key provided.
Then, establish the base habitat:
- Set up base habitat:
sudo yum install git vim htop zsh xeyes
- Get oh-my-zsh:
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"
- Change bash –> zsh:
sudo chsh -s $(which zsh) $(whoami)
- Tinker with
- (Optional): arrange for the return of the Sage Cow
Step 2: system packages
We need to get them pesky system libraries installed. Takes a bit of tinkering, details vary on the individual Linux AMI details:
- Install this:
yum groupinstall "Development tools"
- More packages to avoid the unfortunate pandas catastrophe of October 10th:
sudo yum install zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel xz xz-devel libffi-devel
- Enable an occasional X11-forwarding (self-note: ahem, on a headless system, why?!):
sudo yum install xauth
- pyenv and OpenCV need this:
sudo yum install openssl-devel readline-devel zlib-devel sqlite-devel mesa-libGL
Step 3: pyenv
sudo yum install pyenv echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.zshrc pyenv install 3.8.5 echo 'pyenv global 3.8.5' >> ~/.zshrc
Step π: vim + ale + pylint = IDE
Add the following as .vimrc:
syntax on colorscheme desert set number set hlsearch set ruler " indentations: filetype plugin indent on " show existing tab with 4 spaces width set tabstop=4 " when indenting with '>', use 4 spaces width set shiftwidth=4 " On pressing tab, insert 4 spaces set expandtab " stop yelling at me VIM! cnoreabbrev W w " ukrainian vim "nmap Ж : "nmap Жй :q "nmap Жц :w " highlight trailing whitespaces highlight ExtraWhitespace ctermbg=magenta guibg=magenta match ExtraWhitespace /\s\+$/
Then run the following to install pylint / python language server (do I really need that [all] tag?) and set up ale:
pip install pylint 'python-language-server[all]' mkdir -p ~/.vim/pack/git-plugins/start git clone --depth 1 https://github.com/dense-analysis/ale.git ~/.vim/pack/git-plugins/start/ale
Add the following to your .pylintrc file to catch those pesky NumPy / PyTorch false-positive errors in pylint:
[TYPECHECK] # List of members which are set dynamically and missed by pylint inference # system, and so shouldn't trigger E1101 when accessed. Python regular # expressions are accepted. generated-members=numpy.*, torch.*
Step 4: let the spice flow
Press the big red
pip install matplotlib numpy scipy ipython jupyter opencv-python and pray I remembered to list all the libs dependencies above.
Step 5: run the notebook server
screen to keep it running even if the ssh connection breaks down:
screen -S jupyter
jupyter notebook --no-browser --port=8888
A followed by
D to detach from screen
Step 6: connect from remote
Establish an ssh connection with port forwarding (remember open the proper ports in AWS security groups):
ssh -i ~/.ssh/ec2-access-key.pem -L 8000:localhost:8888 firstname.lastname@example.org
Then connect to the running notebook server in the browser!
Step I’ve-lost-count: start actually working!
See, it was, ahem, very easy! So next time,
just use Ubuntu image just make a Docker container already be stubborn and keep installing out-of-scope stuff on a headless machine for no apparent reason.