Most important bash commands for managing processes, Git, Python, R, SQL/SQLite and LaTeX for researchers and data scientists. This cheat sheet only focusses on bash commands run from the terminal.
The following cheat sheet for commonly used Python Turtle commands will get you up and running with Python Turtle quickly. Turtle is a fun program that dates all the way back to the 1960s when Seymour Papert and his colleagues at MIT created the programming language LOGO which could control a robot turtle with a physical pen in it. Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
Table of Contents
It is an interactive cheat sheet that brings the answer to you. It is a really simple idea but it worked so well that it has become an indispensable tool for me any time I'm coding. Type in what you are looking for in the search bar and the speed sheet will display the answer. Scrapy cheatsheet. GitHub Gist: instantly share code, notes, and snippets.
- Managing processes
- First-aid procedure for killing a running process
- Clone repository from GitHub to local machine
- Virtual environments
- Open new window
- SQL and SQLite
- Repair corrupt database
- Text editing and LaTeX
- Calculate the number of words in a Latex file
First-aid procedure for killing a running process
- open new terminal window
ps + enter
- identify PID (processid) of the process
kill -9 <PID>
control + C(twice if needed)
Cronjobs and Crontab
Schedule crontab task
- If you want to run the cronjob on a server: enter the server
crontab -ein the terminal
<minutes> <hours> <day of month> <month> <day of week>
- for example
6 0 * * 1-6 cd /home/annerose/Python/continuousscraper/ && python processcontrol.py
- this signifies that the process will start to run Monday through Saturday at 6 minutes past midnight.
For more information, see
- http://www.everydaylinuxuser.com/2014/10/an-everyday-linux-user-guide-to.html and
Kill an existing cronjob
ps -ein the terminal to see all existing processes.
- determine which processid your process has.
kill -9 <processid>
Tmux allows to keep processes running after ending an ssh session. For more detailed explanation, see here.
- ssh into the remote machine
- start tmux by typing
tmuxinto the shell
- start the process you want inside the started tmux session
- leave/detach the tmux session by typing
You can now safely logoff from the remote machine, your process will keep running inside tmux. When you come back again and want to check the status of your process you can use
tmux attach to attach to your tmux session.
If you want to have multiple session running side-by-side you should name each session using
$. You can get a list of the currently running sessions using
Some more useful tmux commands (see also this video):
|to tell the shell that it’s for tmux and not just normal shell.|
|split window vertically into two parts|
|split-horizontally : split window horizontally|
|create a new tmux session|
|close (kill) tmux pane|
|detach from tmux session. (without stopping the process)|
|List all tmux sessions|
|attach to a certain tmux session|
|attach all tmux sessions/ any tmux session|
Create bash profile
touch creates the file, so no need to run this command when the file already exists. Alternative:
For editing the .bash_profile. opens in a text editor. See here
Clone repository from GitHub to local machine
- create new repository on GitHub
- go to the directory on your local machine where the cloned repository should be saved.
git clone https://github.com/your-name/repository-name.git
- the repository should now appear in the local folder on your machine.
Commit file from terminal
- go to the directory of your repository inside the terminal
git add .This recurses into sub-directories. Alternative:
git commit -a
git commit -m “your commit message”. Commit the changes.
git push. Push the changes.
To see the status of your repository:
See this useful blog.
Branches are very important when you collaboratively work on Github.
This github page contains useful information on how to create a new branch and how to manage branches on github.
- go to the directory of your repository inside the terminal
- before creating a new branch, make sure all changes are pulled to your local repository
- Create new branch by typing
git checkout -b [name_of_your_new_branch]
- Push the new branch to github by typing
git push origin [name_of_your_new_branch]
- Check out which branches exist for this repository:
git branch. (If there is only the master branch, it will return
- Add a new remote for your branch:
git remote add [name_of_your_remote]. A remote (URL) is Git’s fancy way of saying “the place where your code is stored.” (see here)
- Push changes from your commit into your branch (= into your remote):
git push [name_of_your_new_remote] [name_of_your_branch]
- Update your branch from the original (master) branch:
git fetch [name_of_your_remote]
- To merge changes between your branch and the original (master) branch, you should first switch to master branch in your terminal:
git checkout master. Then simply type
git merge [name_of_your_branch].
global .gitignore file
Create a global .gitignore file (file types to be excluded from every git project):
The file is found under Documents/Username (as a hidden file). Open it in a text editor to edit it and add files you don’t want tosync with git/GitHub.
local .gitignore file
In the terminal, go to the working directory of the project you want to commit to github.
The file is found locally in the working environment of the project. Open it in a text editor to edit it and add files.
How to prevent conflicts in a collaborative Github project
The following procedure should help you considerably to prevent conflicts in collaborative Github and Git project.
Before you start working: pull
Once you’ve made any changes to the project:
- If you get an error message, clean the file, solve conflicts
To summarize: pull, commit, pull, clean, push
Solve conflict using VIM editor
See this Stackoverflow post: http://stackoverflow.com/questions/5599122/problems-with-entering-git-commit-message-with-vim
MacOS Catalina updates improve the stability, performance, or compatibility of your Mac and are recommended for all Catalina users. To get these updates, choose System Preferences from the Apple menu , then click Software Update. Learn more about updating the software on your Mac. Update mac os catalina.
If there is a conflict between your local version of the project and the version on Github, a window of the VIM editor will open after you’ve tried to commit your local changes. In this case, you should proceed as follows:
iinto the VIM editor, which opens the editing (insert”) mode
- type your merge message
Escto be sure to have left insert mode
- then type
Enter, which writes the current file and then closes it.
- your merge should now have been accepted.
Push commits from terminal with two-factor authentification
See this helpful page on how to push commits from the terminal when using two-factor authentification on Github:
Important: You need to use your personal access token, not your Github password to push commits from the terminal.
Change virtual environment:
How to set up and manage virtual environments in Ubuntu: http://askubuntu.com/questions/244641/how-to-set-up-and-use-a-virtual-python-environment-in-ubuntu
Configure Pycharm to use a virtual environment
Then set the shell Preferences->Tools->Terminal->Shell path to
/bin/bash --rcfile ~/.pycharmrc
Check which python packages are installed
Start scrapy project
Python Scrapy Cheat Sheet
Start scrapy project for webscraping: enter the following commandin the terminal (in the directory where you want to start your project).
Open new window
Python Scrapy Cheat Sheet
- Open new RStudio window from terminal (e.g. when one RStudio needs to run for an extended period of time):
open -n -a 'rstudio'in terminal
- How to add an RStudio project to Github: https://www.r-bloggers.com/rstudio-and-github/
Add R project to Github
Add the following commands in shell after having created the project in Github:
Markdown and R
Render/compile an R Markdown file from Terminal:
This resource on R Markdown is helpful.
An R Markdown cheatsheet is available from RStudio here.
Scrapy Cheat Sheet 2020
Set options, even options that aren’t defined by default. This can be useful for example for setting your consumer key, consumer secret etc. of your Twitter app:
SQL and SQLite
Repair corrupt database
How to repair db database: see stackoverflow
Merge two SQLite databases
Scrapy Xpath Cheat Sheet
Leaving out duplicates:
Open new SQLiteBrowser window from terminal
SQLiteBrowser is well suited for viewing and editing database files compatible with SQLite.
If you want to view several databases side by side, you have to open a new SQLiteBrowser window from terminal (it doesn’t seem to be possible to open a new window from within SQLiteBrowser). To this end, go to the directory where yourapplications are stored (in Mac). Normally, this should be:
Thereafter, type the command to open a new SQLiteBrowser window:
Text editing and LaTeX
Calculate the number of words in a Latex file
- Change the working directory of your terminal to where the LaTeX TeX file is located.
- I use one of the two options: (1)
detex <document_name>.tex wc -w -c -lor just
detex <document_name>.tex wc
To calculate word count in pdf document:
pdftotext <document_name>.pdf - wc -w
texcount -1 <document_name>.tex
There are thousands of options for
For example, for including the bibliography in the word count, use
texcount -1 -incbib <document_name>.tex
To include several documents in the word count (e.g. main paper and appendix), just add the different documents behind one another:
texcount -1 -incbib <main_document>.tex <appendix>.tex
For more information on
texcount, see this website