Installing Jupyter Notebook¶
In [595]:
using Pkg
Pkg.add("IJulia")
using IJulia
#notebook()
The notebook interface¶
List of commands in the command palette¶
Ctrl + Shift + p or Command + Shift + p
Help¶
Navigation bar -> Help or press h if in Edit mode
Cells¶
- Code cell, contains code to be executed
- Markdown cell for text formatting
- Use arrow keys to navigate up and down cells
- Hit Return to enter into cell in coding mode or
- press m and press Return to enter into markdown mode
- Execute cell by pressing run in navigation bar or
- press Ctrl + Return or Shift + Return
- Press Esc to leave cell editing
Activating your enviroment¶
In [554]:
using Pkg
Pkg.activate(".")
In [555]:
f = open("output.txt", "w")
Out[555]:
In [556]:
write(f, "This is a first line of text.\n")
Out[556]:
In [557]:
write(f, "This is another line of text.\n")
Out[557]:
In [558]:
close(f)
Many things can go wrong when reading or writing files, therfore it is good practise
to catch possible errors and to make sure the filestream is closed again:
In [559]:
f = open("output.txt", "r")
try
l = readline(f)
l = readline(f)
write(f, "last line of text.\n")
catch exc
println("Something went wrong: $exc")
finally
close(f)
end
Another way of opening a stream for writing:
In [560]:
open("other_output.txt", "w") do f
write(f, "text for another file.")
end
Out[560]:
JSON¶
Structure for universal data exchange.
In [561]:
Pkg.add("JSON");
using JSON
Read and write JSON data:¶
Create sample dictionary to hold the data
In [562]:
# dictionary to write
dict1 = Dict("key1" => 1, "key2" => 2,
"key3" => Dict("sub1"=>"some text","sub2"=>5.4335,"sub3"=>[3,5,7,3]))
Out[562]:
In [563]:
# cast dict into json format
stringdata = JSON.json(dict1)
Out[563]:
Writing text string to output stream:¶
In [564]:
f = open("out_json.json", "w")
try
write(f, stringdata)
catch exc
println("Something went wrong: $exc")
finally
close(f)
end
Out[564]:
Reading back the JSON file:¶
In [565]:
jDict = Dict()
open("out_json.json", "r") do f
global jDict
jDict=JSON.parse(f) # parse and transform data
end
Out[565]:
For other IO (Multimedia, Network) there is extensive documentation:
https://docs.julialang.org/en/v1/base/io-network/
Installing a few more Packages¶
In [566]:
Pkg.add("CSV");
Pkg.add("XLSX");
Pkg.add("DataFrames");
In [567]:
using CSV
using DataFrames
using XLSX
Loading Excel Data¶
In [568]:
xf = XLSX.readxlsx("demo_data.xlsx")
Out[568]:
In [569]:
XLSX.sheetnames(xf)
Out[569]:
In [570]:
sh = xf["Sheet1"]
typeof(sh)
Out[570]:
Selecting all data from Sheet
In [571]:
sh[:]
Out[571]:
In [572]:
# Selecting only 2 rows and 2 columns
sh["A1:B3"]
Out[572]:
Lets cast the Matrix into a DataFrame
In [573]:
# Leave the first row out (column names)
hr = sh["A2:C6"] |> DataFrame
Out[573]:
In [574]:
# Manually rename the columns
rename!(hr, [:Name, :Salary, :Entry_Date])
Out[574]:
We can combine all these steps into one:
In [575]:
hr = DataFrame(XLSX.readtable("demo_data.xlsx", "Sheet1")...)
Out[575]:
In [576]:
typeof(hr)
Out[576]:
Let's save the dataframe as CSV file:¶
In [577]:
CSV.write("HR_Sheet.csv", hr)
Out[577]:
Read in CSV file:¶
In [578]:
hr_csv = CSV.File("HR_Sheet.csv")
Out[578]:
In [579]:
# Select the Name field of the first CSV row
hr_csv[1][:Name]
Out[579]:
Use DataFrames instead of CSV.Row¶
In [580]:
hr_csv = CSV.File("HR_Sheet.csv", header=true) |> DataFrame
Out[580]:
Now, let's read another Excel file:¶
In [581]:
hr_departments = DataFrame(XLSX.readtable("demo_data2.xlsx", "Sheet1")...)
Out[581]:
Casting into datatypes while reading:¶
In [511]:
hr_departments = DataFrame(XLSX.readtable("demo_data2.xlsx", "Sheet1", infer_eltypes=true)...)
Out[511]:
Let's join both tables into one:
In [512]:
hr_all = innerjoin(hr_csv, hr_departments, on=:Name)
Out[512]:
Selecting fields and sub-sets¶
Selecting a single column:
In [513]:
hr_all.Name
Out[513]:
In [514]:
hr_all[!, :Name]
Out[514]:
Sub-setting by condition:¶
In [515]:
# Our condition:
hr_all.Salary .> 75000
Out[515]:
In [516]:
hr_all[hr_all.Salary .> 75000, :]
Out[516]:
Sub-setting columns by name:¶
In [519]:
hr_all[!, [:Name, :Entry]]
Out[519]:
Using the Query.jl package¶
In [520]:
Pkg.add("Query");
using Query;
using Statistics
Pipe operator "|>" is used to feed the output of an operation into the input of the next function, e.g. select, filter and sub-set dataframes:
The command from above:
hr_all[hr_all.Salary .> 75000, :]
can be written as:
In [521]:
hr_all |> @filter(_.Salary > 75000)
Out[521]:
This way we can pipe easily multiple commands:
In [522]:
top = hr_all |>
@select(:Name, :Salary, :Department) |>
@filter(_.Salary > 75000) |>
DataFrame
Out[522]:
In [523]:
df2 = hr_all |>
@groupby(_.Department) |>
@map({Department=key(_), Salary=mean(_.Salary)}) |>
DataFrame
Out[523]:
Detailed description of parameters for import and export can be found in the package documentation:
CSV.jl¶
https://csv.juliadata.org/stable/
XLSX.jl¶
https://felipenoris.github.io/XLSX.jl/stable/
For example in CSV.jl you can normalize the column names to remove unwanted characters
as whitespaces etc.
How to download data from url:¶
In [582]:
d=download("https://raw.githubusercontent.com/fivethirtyeight/data/master/us-weather-history/KCLT.csv")
Out[582]:
In [583]:
weather = CSV.File(d) |> DataFrame
describe(weather)
Out[583]:
Iris Dataset¶
- sepal length in cm
- sepal width in cm
- petal length in cm
- petal width in cm
- class: -- Iris Setosa -- Iris Versicolour -- Iris Virginica
In [584]:
Pkg.add("RDatasets");
using RDatasets;
In [585]:
# Loading the Iris dataset:
iris = dataset("datasets", "iris");
size(iris)
Out[585]:
Explore the table by using first(<data>, nrows)
In [586]:
first(iris, 3)
Out[586]:
In [587]:
# Using sclicing
iris[1:3, :]
Out[587]:
In [588]:
describe(iris)
Out[588]:
Plotting data¶
using Plots.jl
In [589]:
Pkg.add("Plots")
using Plots
Basic line plot:¶
In [590]:
plot(1:10, rand(10))
Out[590]:
we can add more data:¶
In [591]:
plot!(rand(10).+1)
Out[591]:
In [592]:
plot([sin,cos], 0:0.1:Ï€)
Out[592]:
Plots with annotations:¶
In [593]:
x = 1:10
y = rand(10, 2)
plot(x, y, title = "Two Lines", label = ["Line 1" "Line 2"], lw = 3)
Out[593]:
Selecting different the plotting backends¶
Plot.jl uses a default backend for plotting
- GR (fast)
- Plotly (interactive)
- Pyplot
- etc
Changing the plotting backend¶
Switch to interactive plotting:
In [537]:
plotly()
Out[537]:
In [538]:
plot(x, y, title = "Two Lines", label = ["Line 1" "Line 2"], lw = 3)
Out[538]:
Adding layout and combining different plots:¶
In [539]:
gr()
p1 = plot(x, y, xlabel = "X Axis", title = "Lineplot") # Line plot
p2 = scatter(x, y) # Scatter plot
p3 = plot(x, y, lw = 3) # Another line plot
p4 = histogram(x, y) # Histogram
plot(p1, p2, p3, p4, layout = (2, 2), legend = false)
Out[539]:
Using the StatsPlots package¶
In [540]:
Pkg.add("StatsPlots")
using StatsPlots
Moving back to the Iris dataset:¶
Scatter plot the petal length vs. petal width
In [541]:
@df iris scatter(:PetalLength, :PetalWidth)
Out[541]:
With StatsPlots we can do a correlation plot of selected parameters:¶
In [542]:
@df iris corrplot([:SepalLength :SepalWidth :PetalLength :PetalWidth], grid = false)
Out[542]:
Histogram of sepal length grouped by species¶
In [543]:
@df iris groupedhist(:SepalLength, group = :Species, bar_position = :dodge)
Out[543]:
Scratching the surface in 45 minutes¶
All packages in Julia are very well documented
- CSV.jl
- XLSX.jl
- JSON.jl
- DataFrames.jl
- Plots.jl
- StatsPlots.jl