Fibinse Xavier fx86

Scraper of all nice data. Data scientist. Saas-ing it.

fx86 / common.py

Last active December 13, 2021 02:56

Necessary code snippets for deep learning #DL

	import gc

	# clear memory in gpu usage and with garbage collection.
	# additionally, prints variables with high memory usage.
	def clear_memory():
	gc.collect()
	torch.cuda.empty_cache()
	''' by Fred Cirera, https://stackoverflow.com/a/1094933/1870254, modified'''
	def sizeof_fmt(num, suffix='B'):
	for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:

fx86 / Object-identification-for-LIDAR

Last active June 4, 2021 10:16

Object identification methods for LiDAR point clouds

	Libraries:
	- [SFA3d](https://github.com/maudzung/SFA3D)
	- Laspy (library for reading LiDAR data)
	- [Pypetree](http://cjauvin.github.io/pypetree/)
	- https://pypi.org/project/lidar/

	Related links
	- What is point cloud data ? https://en.wikipedia.org/wiki/Point_cloud
	- https://towardsdatascience.com/point-cloud-data-simple-approach-f3855fdc08f5
	- [(Medium) Essentials of LiDAR data processing](https://geospatialawarenesshub.com/blog/essentials-of-lidar-point-cloud-processing-using-python/)

fx86 / keybase.md

Last active November 12, 2019 02:09

I hereby claim:

To claim this, I am signing this object:

fx86 / fresh-install.sh

Last active June 4, 2021 10:16

A bash script to make MacBook ready for data science

	# install brew
	ruby -e "$(curl -fsSL https://raw.zshhubusercontent.com/Homebrew/install/master/install)"

	brew doctor

	# install zsh
	brew install zsh

	# install wget using brew
	brew install wget

fx86 / query_redash.py

Last active July 27, 2018 06:26

Takes a Redash query ID and returns cached-data in a Pandas dataframe

	# save your redash API key in a text file as below
	api_key = open('api_key.config', 'r').readlines()[0]



	def api_to_df(query_id, api_key=api_key):
	'''
	Takes a Redash query ID and returns cached-results
	in a Pandas dataframe
	'''

fx86 / boilerplate.py

Last active June 24, 2018 07:32

Boilerplate code for scrapers - has query-caching, geocoding, refreshing proxies & enabling detailed logging

	from glob import glob
	from hashlib import md5
	from fake_useragent import UserAgent as ua
	from splinter import Browser
	from splinter.exceptions import *
	from selenium.common.exceptions import *
	from random import randint
	from time import sleep
	import requests
	import time

fx86 / string_similarity.py

Created May 28, 2016 11:15

A much more effective algorithm for string similarity found on Stack Overflow

	import wikipedia

	def get_bigrams(string):
	'''
	Takes a string and returns a list of bigrams
	'''
	s = string.lower()
	return [s[i:i+2] for i in xrange(len(s) - 1)]

	def string_similarity(str1, str2):