Jekyll2018-11-27T01:12:42+01:00https://blog.undergrinder.com/undergrinder’s blogTech blog for metalheadsundergrinderExcel String Tricks2018-11-26T00:00:00+01:002018-11-26T20:05:48+01:00https://blog.undergrinder.com/tutorial/2018/11/26/excel-string-tricks<p><img src="\assets\20181126\stringtricks.png" alt="stringTricks" /></p>
<!--more-->
<p>This time I would like to show you some cool tricks that you can do in Excel. The post is inspired by <a href="https://chandoo.org/wp/extract-file-name-from-full-path-using-formulas">Chandoo’s awesome blogpost</a> about how to get the file name from full path, specifying how to get the last portion of a string value. Chandoo’s post includes several good trick, worth it to check it. I provide an other kind of approach to achieve the same, only with formulas, at the end of the post you’ll be able to perform these kind of string manipulation in Excel + you’ll understand it (I hope at least). Example workbook can be founded <a href="\assets\20181126\stringtricks.xlsx">here</a>.</p>
<p>If you use Excel in other language than english try the <a href="https://hu.excel-translator.de/translator">Excel translator</a>.</p>
<h1 id="excel-string-trick-1---how-many-characterstring-portion-do-we-have-in-a-cell">Excel String Trick 1. - How many character/string portion do we have in a cell?</h1>
<p>This example we will find out how many occurrences have the quote below.</p>
<blockquote>
<p>All Dog Breeds, All Dog Types, All Dog List Names & Pictures</p>
</blockquote>
<h2 id="wrong-approaches">Wrong approaches</h2>
<p>Let’s say the example is in <em>A1</em>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=countif(A1:A1,'dog')
</code></pre></div></div>
<p>The result is <strong>0</strong>, because countif try to match exactly.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=countif(A1:A1,'*dog*')
</code></pre></div></div>
<p>The result is <strong>1</strong>, because the formula uses wildcard character, but if we evaluate it right we learned, that the cell contains at least one the characters ‘dog’. It is a good method to create flags or something.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=search('dog',A1)
</code></pre></div></div>
<p>The result is <strong>5</strong>, because the <em>search</em> function return the index of the first character of our searchText. Be aware this function is case insensitive.</p>
<h2 id="the-solution">The solution</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=(len(A1)-len(substitute(lower(A1),"dog","")))/len("dog")
</code></pre></div></div>
<p>For readers, who use Excel in hungarian, and also are lazy to use <a href="https://hu.excel-translator.de/translator">Excel translator</a>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=(HOSSZ(A1)-HOSSZ(HELYETTE(KISBETŰ(A1);"dog";"")))/HOSSZ("dog")
</code></pre></div></div>
<p>The solution is a bit tricky, let’s see it step by step</p>
<table>
<thead>
<tr>
<th>Step</th>
<th>Formula snippet</th>
<th>Explanation</th>
<th>Result</th>
</tr>
</thead>
<tbody>
<tr>
<td>1.</td>
<td>len(A1)</td>
<td>Get the length of the original string</td>
<td>60</td>
</tr>
<tr>
<td>2.1</td>
<td>lower(A1)</td>
<td>Make the text case insensitive with lower (or upper)</td>
<td> </td>
</tr>
<tr>
<td>2.2</td>
<td>substitute(lower(A1);”dog”;””)</td>
<td>Substract the searched characters</td>
<td> </td>
</tr>
<tr>
<td>2.3</td>
<td>len(substitute(lower(A1),”dog”,””))</td>
<td>Get the length of the text without the searched characters</td>
<td>51</td>
</tr>
<tr>
<td>4.</td>
<td>len(A1)-len(substitute(lower(A1),”dog”,””))</td>
<td>Substract the two length from each other</td>
<td>9</td>
</tr>
<tr>
<td>5.</td>
<td>len(“dog”)</td>
<td>Get the length of our search text.</td>
<td>3</td>
</tr>
<tr>
<td>6.</td>
<td>=(len(A1)-len(substitute(lower(A1),”dog”,””)))/len(“dog”)</td>
<td>All in one formula</td>
<td>3</td>
</tr>
</tbody>
</table>
<p>With this method we can easily count any character(s) occurences in any string.</p>
<h1 id="excel-string-trick-2---extract-file-name-from-path">Excel String Trick 2. - Extract file name from path</h1>
<p>As in intro mentioned I would like to show you an other method to get file name from path than <a href="https://chandoo.org/wp/extract-file-name-from-full-path-using-formulas">Chandoo</a> (Really smart trick).</p>
<h2 id="1-get-file-names">1. Get file names</h2>
<p>Open a cmd (or type the command line or <a href="https://www.ghisler.com">Total Commander</a>) in a specific directory and execute the command below in a specific folder in order to get the file names with fullpath ( /b option is for simple or bare format)</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dir /b|clip.exe
</code></pre></div></div>
<p>This command pipelining the folder content to the clipboard, simple press a ctrl+v after the command execution in you excel worksheet.</p>
<p>If you want to get the subdirectories as well, then (/s for subdirectories…)</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dir /b /s|clip.exe
</code></pre></div></div>
<p>or you can redirect the output to a csv file</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dir /b /s>files.csv
</code></pre></div></div>
<p>or powershell</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">Get-Childitem</span> -recurse<span class="o">)</span>.fullname|clip.exe
</code></pre></div></div>
<p>You can use any of <em>get-childitem</em>’s aliases like *dir; ls; gci * etc…</p>
<h2 id="2-use-excel-string-trick-1-to-get-how-many-backslash-are-in-paths-there">2. Use Excel String Trick 1. to get how many backslash are in paths there</h2>
<p>Example string:</p>
<blockquote>
<p>F:\blog\_posts\2018-11-26-excel-string-tricks.md</p>
</blockquote>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=(LEN(A2)-LEN(SUBSTITUTE(A1,"\","")))
</code></pre></div></div>
<p>Notice it we don’t need divide the result because the search string is just one character.</p>
<p><strong>Result: 3</strong></p>
<h2 id="3-locate-the-last-backslash">3. Locate the last backslash</h2>
<p>To get the filename, we need everything after the last backslash. The <em>substitue</em> function has an optional parameter, where we can specify which occurrence would like to replace with other character.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=SUBSTITUTE(A1,"\","@",LEN(A1)-LEN(SUBSTITUTE(A1,"\","")))
</code></pre></div></div>
<p>The optional parameter is our pervious point, with this we changed the last “\” to “@”. You can choose any other character that not present in the string.</p>
<p><strong>Result: ‘<em>F:\blog\_posts@2018-11-26-excel-string-tricks.md</em></strong>’</p>
<p>We can easily locate this newly replaced character</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=SEARCH("@",SUBSTITUTE(A7,"\","@",LEN(A7)-LEN(SUBSTITUTE(A7,"\",""))))
</code></pre></div></div>
<p><strong>Result: 15</strong></p>
<h2 id="4-get-the-file-name--all-in-one">4. Get the file name + All in one</h2>
<p>So far we know the filename start after the 15th character, so we need to find out how many characters left right to it</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=LEN(A1)-SEARCH("@",SUBSTITUTE(A7,"\","@",LEN(A7)-LEN(SUBSTITUTE(A7,"\",""))))
</code></pre></div></div>
<p><strong>Result: 33</strong></p>
<p>Get X (=previous result) character from right to get the filename</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=RIGHT(A2,LEN(A2)-SEARCH("@",SUBSTITUTE(A2,"\","@",LEN(A2)-LEN(SUBSTITUTE(A2,"\","")))))
</code></pre></div></div>
<p><strong>Result: ‘<em>2018-11-26-excel-string-tricks.md</em>‘</strong></p>
<p>The example work book is <a href="\assets\20181126\stringtricks.xlsx">here</a>, please leave a comment if you liked this content or whatever :)</p>
<p>:metal: :poop: :metal:</p>undergrinderLet there be Dark….2018-04-26T00:00:00+02:002018-04-27T21:05:48+02:00https://blog.undergrinder.com/tutorial/2018/04/26/let-there-be-dark<p>…or how to create a cool display darkener.</p>
<p><img src="/assets/20180427/1.jpg" alt="Let there be dark" title="Daaaark!!" /></p>
<!--more-->
<p>So today we make our screen off just with one click on that icon:</p>
<p><img src="/assets/20180427/2.png" alt="cool icon" title="Desktop icon" /></p>
<p>With this little utility we can control the screen easily. It is better than set too little time to auto screen off.</p>
<h1 id="create-a-blueprint">Create a blueprint</h1>
<p>First we create a blueprint, that we pimp up later. Create a new file and make its extension to <code class="highlighter-rouge">.bat</code></p>
<p><img src="/assets/20180427/3.png" alt="create batch" title="create batch" /></p>
<p>Open it and type the following lines</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@echo off
cls
color 0
title Let there be dark
ECHO <span class="k">**********************************************************</span>
ECHO <span class="k">*</span> Fear of the dark <span class="k">*</span>
ECHO <span class="k">**********************************************************</span>
ECHO.
ECHO Screen off......
ECHO.
</code></pre></div></div>
<p>Feel free to change the title, color and so on…</p>
<h1 id="choose-the-method">Choose the method</h1>
<h2 id="the-ugly---set-empty-screensaver">The ugly - set empty screensaver</h2>
<p>Append the following line to our blueprint</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>REM start en empty screensaver
scrnsave.scr /s
</code></pre></div></div>
<p>After save, we are already control our screen. But it has one major pitfall. The screen is not turned off really, just dark, but there is light as well. It is okay for those, who fear of the dark…</p>
<p><img src="/assets/20180427/4.png" alt="fear of the dark" title="fear of the dark" /></p>
<h2 id="the-less-ugly--3rd-party-tool">The less ugly- 3rd party tool</h2>
<p>If we want a more sophisticated result then check <a href="http://www.nirsoft.net/utils/nircmd.html">NirCMD</a> utility. It has a lot of fancy function with you can control the computer, without deep knowledge of the soul of your hardware and operating system.</p>
<p><img src="/assets/20180427/5.png" alt="Add NirCMD" title="NirCMD" /></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nircmd monitor off
</code></pre></div></div>
<h1 id="a-little-cosmetic">A little cosmetic</h1>
<p>When we move the mouse the darkness gone, during night it feels like armageddon. So we customize our batch further.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:darkness
nircmd monitor off
ECHO %incr% RE-darking... Press any key %date% %time%
SET /A <span class="nv">incr</span><span class="o">=</span>%incr%+1
PAUSE <span class="o">></span>nul
GOTO :darkness
</code></pre></div></div>
<p>We create loop with labels, when the screen turn on, with keypress we GO TO :darkness label again till the end of time.</p>
<p><img src="/assets/20180427/6.png" alt="in operation" title="in operation" /></p>
<h1 id="create-a-shortcut-icon">Create a shortcut icon</h1>
<p>Create a shortcut icon and place on your desktop. Choose a fancy icon for it, I prefer the dark clock for this task.</p>
<p><img src="/assets/20180427/2.png" alt="cool icon" title="Desktop icon" /></p>
<p>Yeah, that’s really cool 😎</p>
<hr />
<p>The full batch file is downloadable from <a href="https://github.com/undergrinder/Scripts-examples/blob/master/Batch/crt-darkness.bat">here</a></p>undergrinder…or how to create a cool display darkener.Ingest the web - Part 02018-04-24T00:00:00+02:002018-04-25T00:55:48+02:00https://blog.undergrinder.com/tutorial/2018/04/24/ingest-the%20web-part0<p>Have you ever dreamed about scraping the <a href="http://www.shockmagazin.hu">shockmagazin.hu</a> and load it’s album reviews to PostgreSQL?
This post is for you.
<img src="/assets/20180424/swallow-muk.png" alt="Ingestion" title="Pokemon swallow each other" /></p>
<!--more-->
<h1 id="steps---short-version">Steps - Short version</h1>
<h2 id="step-1">Step 1.</h2>
<p><img src="/assets/20180424/shockmagaizn.png" alt="shockmagazin" title="shockmagazin" /></p>
<h2 id="step-2">Step 2.</h2>
<p><img src="/assets/20180424/magic.gif" alt="magic" title="magic" /></p>
<h2 id="step-3">Step 3.</h2>
<p><img src="/assets/20180424/postgres.png" alt="postgres" title="postgres" /></p>
<h1 id="steps---longer-version">Steps - longer version</h1>
<h2 id="step-1-1">Step 1.</h2>
<p>Let’s see our source <a href="http://www.shockmagazin.hu/cd-kritika/blog">shockmagazin.hu - CD reviews</a>, and examine a section’s URL.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code>http://www.shockmagazin.hu/component/abc/?view=alphabetical<span class="err">&</span>task=A<span class="err">&</span>sectionid=14<span class="err">&</span>catid=77
</code></pre></div></div>
<p>As we can see the categories are appearing as URL parameter, so if we had a list from categories then we would be happy.</p>
<p><img src="/assets/20180424/categorylist_shock.png" alt="shockmagazin categories" title="shockmagazin category list" /></p>
<p>*The entirely pshell script is available on <a href="https://github.com/undergrinder/Scripts-examples/blob/master/PowerShell/shockDownloader.ps1">Github (❤️)</a></p>
<h3 id="step-11---define-the-categories">Step 1.1 - Define the categories</h3>
<p>I chose powershell for webscraping, because I am familiar with it, but your decision can be differ from it for example <a href="http://phantomjs.org">phantomJS</a> <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> or python, there are enourmous number of utility and method.</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$urlCategory</span> <span class="o">=</span> @<span class="o">(</span><span class="s1">'0-9'</span>,<span class="s1">'A'</span>,<span class="s1">'B'</span>,<span class="s1">'C'</span>,<span class="s1">'D'</span>,<span class="s1">'E'</span>,<span class="s1">'F'</span>,<span class="s1">'G'</span>,<span class="s1">'H'</span>,<span class="s1">'I'</span>,<span class="s1">'J'</span>,<span class="s1">'K'</span>,<span class="s1">'L'</span>,<span class="s1">'M'</span>,<span class="s1">'N'</span>,<span class="s1">'O'</span>,<span class="s1">'P'</span>,<span class="s1">'Q'</span>,<span class="s1">'R'</span>,<span class="s1">'S'</span>,<span class="s1">'T'</span>,<span class="s1">'U'</span>,<span class="s1">'V'</span>,<span class="s1">'W'</span>,<span class="s1">'X'</span>,<span class="s1">'Y'</span>,<span class="s1">'Z'</span><span class="o">)</span>
<span class="nv">$link</span> <span class="o">=</span> <span class="s1">'http://www.shockmagazin.hu/component/abc/?view=alphabetical&task='</span>
</code></pre></div></div>
<p>We created an array <code class="highlighter-rouge">$urlCategory</code> that holds our category list and a variable <code class="highlighter-rouge">$link</code> with the base url.</p>
<h3 id="step-12---loop-through-links-then-loop-though-links">Step 1.2 - loop through links then loop though links…</h3>
<p>Next we are going to start loop the category list, that holds many links, so we loop the category page as well. Our pseudo code is ought to seem like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lisr[category]
storageForItems
loop in categories
loop in album links
store it's data in storageForItems
if it's an album review
write to disk
</code></pre></div></div>
<p>To implement our pseudo code we use pshell’s <code class="highlighter-rouge">Invoke-Webrequest</code>, this cmdlet returns a .net object with many properties and methods (COOL 😎).</p>
<p>-1. Create the <em>storageForItems</em>, that will hold a custom powershell <code class="highlighter-rouge">object$cdItemArray = @()</code>. It is empty so far.</p>
<ol>
<li>
<p>Start a loop <code class="highlighter-rouge">foreach($urlParam in $urlCategory)</code></p>
</li>
<li>Create the url request <code class="highlighter-rouge">$url = $link+$urlParam</code></li>
<li>Get the data with <em>Invoke-Webrequest</em> cmdlet <code class="highlighter-rouge">$webrequest = invoke-webrequest $url</code></li>
<li>Get the url list. Our <em>$webrequest</em> object has a <em>ParsedHTML</em> property, it’s represent the html document even better we can query it’s content. Since the <em><li></em> tag holds the content we interested in, we query these tags with <em>getElementById</em>, afterward filter this list, so our new <code class="highlighter-rouge">$urlList</code> array holds only the album reviews link.</li>
</ol>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$urlList</span> <span class="o">=</span> <span class="nv">$webrequest</span>.ParsedHtml.body.getElementsByTagName<span class="o">(</span><span class="s1">'li'</span><span class="o">)</span>|foreach<span class="o">{</span><span class="nv">$_</span>.innerhtml<span class="o">}</span>
<span class="nv">$urlList</span> <span class="o">=</span> <span class="nv">$urlList</span>|where-object<span class="o">{</span><span class="nv">$_</span> -like <span class="s1">'*/cd-kritika/*'</span> -and <span class="nv">$_</span> -notlike <span class="s1">'*/cd-kritika/blog*'</span> -and <span class="nv">$_</span>-notlike <span class="s1">'*#comment*'</span><span class="o">}</span>
</code></pre></div></div>
<ol>
<li>Loop again - but this time in album links on the current category page.<img src="/assets/20180424/loopagain.png" alt="loop again" title="loop again" /></li>
</ol>
<h3 id="step-13---comprehend-the-result">Step 1.3 - comprehend the result</h3>
<p>Example<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup></p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><a</span> <span class="na">href=</span><span class="s">"/cd-kritika/cannibal-corpse-torture"</span><span class="nt">></span>Cannibal Corpse: Torture<span class="nt"></a></span>
</code></pre></div></div>
<p>The result is found above, notice it’s contains three kind of information</p>
<ol>
<li>Album URL - between html tags</li>
<li>Band name - at the start of the tag contents</li>
<li>Album title - in the tag content, after the semicolon</li>
</ol>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#main splitting</span>
<span class="nv">$itemDict</span> <span class="o">=</span> <span class="nv">$Item</span> -replace <span class="s1">'.*(href=")'</span>,<span class="s1">''</span>
<span class="nv">$itemDict</span> <span class="o">=</span> <span class="nv">$itemDict</span>.replace<span class="o">(</span><span class="s1">'</A>'</span>,<span class="s1">''</span><span class="o">)</span>
<span class="nv">$itemDict</span> <span class="o">=</span> <span class="nv">$itemDict</span>.split<span class="o">(</span><span class="s1">'>'</span><span class="o">)</span>
<span class="c1">#add the first part to our object</span>
<span class="nv">$cdItem</span>|Add-Member -NotePropertyName url -NotePropertyValue <span class="nv">$itemDict</span><span class="o">[</span>0]
<span class="c1">#seconder split and assign to object</span>
<span class="nv">$cdItem</span>|Add-Member -NotePropertyName band -NotePropertyValue <span class="o">(</span><span class="nv">$itemDict</span><span class="o">[</span>1]<span class="o">)</span>.split<span class="o">(</span><span class="s1">':'</span><span class="o">)[</span>0]
<span class="nv">$cdItem</span>|Add-Member -NotePropertyName album -NotePropertyValue <span class="o">(</span><span class="nv">$itemDict</span><span class="o">[</span>1]<span class="o">)</span>.split<span class="o">(</span><span class="s1">':'</span><span class="o">)[</span>1]
</code></pre></div></div>
<p>Please do not forget this <code class="highlighter-rouge">$cdItem</code> object add to our <code class="highlighter-rouge">$cdItemArray</code> array.</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$cdItemArray</span> <span class="o">=</span> <span class="nv">$cdItemArray</span> + <span class="nv">$cdItem</span>
</code></pre></div></div>
<h3 id="step-14---write-data-to-disk">Step 1.4 - Write data to disk</h3>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$cdItemArray</span>|export-csv shockmagazin.csv -delimiter <span class="s1">';'</span>
</code></pre></div></div>
<p><img src="/assets/20180424/preview_csv.png" alt="loop again" title="preview csv" /></p>
<p>A little tailoring is needed like remove the first line, and trim the quotes, it’s necessity depends on the capabilities of the next processing step’s tool (ignore first x lines, set the text separator etc.)</p>
<p><img src="/assets/20180424/preview_the_run.png" alt="Sample from script execution" title="sample from script execution" /></p>
<p><em>Sample from script execution</em></p>
<h2 id="step-2---skippable">Step 2. - skippable</h2>
<p><img src="/assets/20180424/magic.gif" alt="magic" title="magic" /></p>
<h2 id="step-3---store-our-data-in-database">Step 3. - store our data in database</h2>
<p>The previous section we store a csv file on hard disk, but like a cool guys we like our data in database. I chose <a href="https://www.postgresql.org/">PostgreSQL</a> for this task, because it is open source without any license pitfalls (sure <a href="https://www.sqlite.org">SQLite</a> suits this particular post case, I know, and I ❤️ SQLite)</p>
<p><img src="/assets/20180424/postgres_tg.png" alt="postgres undergrinder's style" title="postgres undergrinder's style" /></p>
<h3 id="step-31---create-table">Step 3.1 - Create table</h3>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">shockinfo</span>
<span class="p">(</span><span class="n">id</span> <span class="n">serial</span> <span class="k">primary</span> <span class="k">key</span><span class="p">,</span>
<span class="n">url</span> <span class="n">character</span> <span class="n">varying</span><span class="p">,</span>
<span class="n">band</span> <span class="n">character</span> <span class="n">varying</span><span class="p">,</span>
<span class="n">album</span> <span class="n">character</span> <span class="n">varying</span><span class="p">);</span>
</code></pre></div></div>
<h3 id="step-32---read-the-csv">Step 3.2 - Read the CSV</h3>
<p>I Assume, that out csv file is well formatted, remoced the first line etc…etc…etc… so we can settle the data into the database, where data should belong to.</p>
<ol>
<li>
<p>Open command line and type <code class="highlighter-rouge">psql -U [youruser]</code>, if the path variable for psql is set correctly, you can type it anywhere in your filesystem</p>
</li>
<li>
<p>In psql execute the right command</p>
</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="se">\c</span>opy shockinfo<span class="o">(</span>url,band,album<span class="o">)</span> from shockmagazin.csv <span class="o">(</span>DELIMITER <span class="s1">';'</span><span class="o">)</span><span class="p">;</span>
</code></pre></div></div>
<p><img src="/assets/20180424/postgres.png" alt="postgres" title="postgres" /></p>
<h1 id="next-steps---homework">Next steps -<em>homework</em></h1>
<p>We gathered and ingest some information from the web. If you are willing to dive deeper and develop the script feel free, it’s downloadable from <a href="https://github.com/undergrinder/Scripts-examples/blob/master/PowerShell/shockDownloader.ps1">Github (❤️)</a>. The next steps can be iterate on album links, and extract information about the albums, e.g. rating, or sniff other additional information from somewhere else, like <a href="https://www.metal-archives.com">metal-archives.com.</a></p>
<hr />
<p>If you liked this post, or stucked/succeced/inspirred/<a href="https://www.metal-archives.com/albums/Depths_of_Depravity/Inspirritation/282791">inspirritationed</a> leave a message below.</p>
<p>Have fun!🐷</p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>It’s a really great tool <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Awesome album <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>undergrinderHave you ever dreamed about scraping the shockmagazin.hu and load it’s album reviews to PostgreSQL? This post is for you.Upcoming concert: Cenobite - Budapest 27-04-20182018-04-02T00:00:00+02:002018-11-26T20:05:48+01:00https://blog.undergrinder.com/misc/2018/04/02/cenobite-0427<p><img src="/assets/20180401/m2.jpg" alt="" title="banner" />
<!--more--></p>
<h1 id="black--death--budapest-s8-underground-club-27-04-2018">Black & Death // Budapest S8 Underground Club, 27-04-2018</h1>
<p>19.30-20.30 <strong>Cenobite</strong><br />
20.45-21.45 Shapes of Distortion (SOD) | Modern Metal, Crossover<br />
22.00-23.00 DunkelNacht [NL-FR] | Black Death Metal<br />
23.15-00.15 Noise Fusion | Metal<br />
00.15-XX.XX Rockdisco till death</p>
<p><img src="/assets/20180401/m1.jpeg" alt="" title="band" /></p>
<p><a href="http://cenobite.hu/">Cenobite</a><br />
<a href="https://www.facebook.com/events/2082616611958196/">Event’s facebook page</a></p>
<h2 id="update---bootleg-from-the-concert">Update - bootleg from the concert</h2>
<iframe width="560" height="315" src="https://www.youtube.com/embed/o9RJaYeXPNU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>undergrinderPimp your Powershell - Create a profile2018-04-01T00:00:00+02:002018-04-01T21:05:48+02:00https://blog.undergrinder.com/powershell/pimp%20your%20stuff/tutorial/2018/04/01/pimp-your-powershell<p>First of all Happy Eastern to everyone! We got some day off because of it so we can dive deeper into
a powershell customization project.
Let’s see, how can we increase thr comfort factor using powershell (referred to as pshell) by tailor our profile. <img src="/assets/20180401/happy_eastern.jpg" alt="happy_eastern" title="Eastern" /></p>
<!--more-->
<p><strong>Table of content</strong></p>
<ul id="markdown-toc">
<li><a href="#what-profile-do" id="markdown-toc-what-profile-do">What profile do</a></li>
<li><a href="#check-your-execution-policy" id="markdown-toc-check-your-execution-policy">Check your Execution Policy</a></li>
<li><a href="#blueprint-of-your-profile-script" id="markdown-toc-blueprint-of-your-profile-script">Blueprint of your profile script</a></li>
<li><a href="#start-customizing" id="markdown-toc-start-customizing">Start customizing</a> <ul>
<li><a href="#greetings" id="markdown-toc-greetings">Greetings</a></li>
<li><a href="#create-our-custom-object-that-makes-the-world-or-at-least-your-session-better" id="markdown-toc-create-our-custom-object-that-makes-the-world-or-at-least-your-session-better">Create our custom object, that makes the world (or at least your session) better!</a></li>
<li><a href="#inform-us-about-the-object" id="markdown-toc-inform-us-about-the-object">Inform us about the object</a></li>
<li><a href="#add-modules" id="markdown-toc-add-modules">Add modules</a> <ul>
<li><a href="#check-what-functioncmdlets-belongs-to-our-profile" id="markdown-toc-check-what-functioncmdlets-belongs-to-our-profile">Check what function/cmdlets belongs to our profile</a></li>
</ul>
</li>
<li><a href="#aliases" id="markdown-toc-aliases">Aliases</a></li>
<li><a href="#drives" id="markdown-toc-drives">Drives</a></li>
<li><a href="#logging-our-session" id="markdown-toc-logging-our-session">Logging our session</a></li>
<li><a href="#godspeed" id="markdown-toc-godspeed">GodSpeed</a></li>
<li><a href="#add-lucy" id="markdown-toc-add-lucy">Add Lucy</a></li>
</ul>
</li>
<li><a href="#bring-all-together" id="markdown-toc-bring-all-together">Bring all together</a></li>
<li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li>
</ul>
<h1 id="what-profile-do">What profile do</h1>
<p>The profile in pshell can be very handy in many situation, for example</p>
<ul>
<li>Declare variables to your session</li>
<li>Load modules</li>
<li>Start logging automatic</li>
<li>Whatever you want</li>
</ul>
<p>So the profile is a script, that runs each time, when the pshell console starts.</p>
<h1 id="check-your-execution-policy">Check your Execution Policy</h1>
<p>Before you can change anything related your profile, you ought to check, which execution policy is set in your machine.</p>
<p>After start, type</p>
<pre><code class="language-Powershell">Get-ExecutionPolicy
</code></pre>
<p>command, the returned string is ‘Restricted’ by great chance. That means we cannot run script inside the shell (I’ll write a workaround in a later post ), so we must change it before start create a profile.
<img src="/assets/20180401/1.png" alt="" title="Execution policy" />
In nutshell there 5+1 kind of these policies, from the strictest to the lenient: <code class="highlighter-rouge">Restricted -> AllSigned -> RemoteSigned -> ByPass -> Unrestricted | +1 Undefined</code>
If you want to read about this policies, check the <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-6">official site</a>.</p>
<p>In order to change the policy you should start a pshell console as administrator
<em>windows+X</em> then select <em>Windows Powershell (Admin)</em> then start typing</p>
<pre><code class="language-Powershell">Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Bypass
</code></pre>
<p>To check the settings, type again
After start, type</p>
<pre><code class="language-Powershell">Get-ExecutionPolicy
</code></pre>
<p><img src="/assets/20180401/2.png" alt="" title="Set-Execution policy" /></p>
<p>The <em>-Scope</em> switch can be omitted, if you want to set the policy for all user, or you can simplify the command to</p>
<pre><code class="language-Powershell">Set-ExecutionPolicy Bypass
</code></pre>
<p>I prefer the verbose mode, it helps a lot later, when you check your scripts.</p>
<h1 id="blueprint-of-your-profile-script">Blueprint of your profile script</h1>
<p>First, find where does live the profile with type <code class="highlighter-rouge">$profile</code>
<img src="/assets/20180401/3.png" alt="" title="Where does live my profile?" />
It will be the <em>C:\Users\undergrinder\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1</em> path, if we customize first time, this file doesn’t exist.
Type <code class="highlighter-rouge">notepad $profile</code> and click <em>yes</em>, if you are asked to create this file in the opened notepad.</p>
<p>I hope your real private directory is not the same with the deafult windows user directory, so we will dot source an other profile file in the official <em>$profile</em> file.
This step can be omitted, and write your code here, I prefer dot sourcing other file. <code class="highlighter-rouge">. D:\undergrinder\real_profile.ps1</code>
Type <code class="highlighter-rouge">notepad D:\undergrinder\real_profile.ps1</code>, or just create it and open with your favorite text editor (mine is <a href="https://notepad-plus-plus.org/download/v7.5.6.html">Notepad++</a>),
and type in the first line <code class="highlighter-rouge">Write-Host 'It WORKS!!!!!' `n -foregroundcolor 'darkgreen'</code></p>
<p><img src="/assets/20180401/4.png" alt="" title="Arrange files" />
When you restart pshell, this message should appear:</p>
<p><img src="/assets/20180401/5.png" alt="" title="Welcome" /></p>
<h1 id="start-customizing">Start customizing</h1>
<h2 id="greetings">Greetings</h2>
<p>I like to see my pshell greet me, so our first line should be:</p>
<pre><code class="language-Powershell">Write-Host 'Hey undergrinder!' `n -foregroundcolor darkgreen
</code></pre>
<h2 id="create-our-custom-object-that-makes-the-world-or-at-least-your-session-better">Create our custom object, that makes the world (or at least your session) better!</h2>
<p>Define an object with regularly used folders, files and so on… The limit is the sky (the limit is powershell’s capabilities…).
This is just an example, feel free to expand/collapse the properties:</p>
<pre><code class="language-Powershell">$tg = new-object psobject `
-property @{home = "e:\undergrinder\home";
desktop = "c:\Users\undergrinder\Desktop";
fav1 = "e:\myFavouriteDirectory1\";
fav2 = "e:\myFavouriteDirectory2\";
modules = "e:\undergrinder\powershell\modules\";
scripts = "e:\myFrequentlyUsedScripts\";
bmode = if([environment]::Is64BitProcess){64} else {32};
startloc = (Get-Location);
transcripts = "e:\undergrinder\powershell\transcripts\";}
</code></pre>
<p>If we wonder what kind of session do we have just type after startup $tg.bmode, and we get 32 if we are in a 32bit session for example.</p>
<h2 id="inform-us-about-the-object">Inform us about the object</h2>
<p>At startup we wants to see which values do we have in our own object, so we list it’s content.</p>
<pre><code class="language-Powershell">Write-host -foregroundcolor 'gray' 'o Session object''s ($tg) elements':
$tg|format-list
</code></pre>
<p><img src="/assets/20180401/6.png" alt="" title="myObject" /></p>
<h2 id="add-modules">Add modules</h2>
<p>Besides the installed modules, we can add modules to the profile as well, I prefer this method in some cases, for example when you star pshell with <em>-noprofile</em> switch
there are modules I don’t want to be loaded into that session.
So we add administrate our modules into our own object:</p>
<pre><code class="language-Powershell">$tg|Add-Member -NotePropertyName loaded_modules `
-NotePropertyValue @()
</code></pre>
<p>We have just added a new property, an array, that will hold our module’s name.
Use the basic <code class="highlighter-rouge">Import-Module</code> cmdlet to import your desired module, but don’t forget to assign it’s name to loaded_modules property.</p>
<pre><code class="language-Powershell">$tg.loaded_modules += 'module1'
</code></pre>
<h3 id="check-what-functioncmdlets-belongs-to-our-profile">Check what function/cmdlets belongs to our profile</h3>
<p>We expand our object again</p>
<pre><code class="language-Powershell">$commands = get-command| `
where-object{$_.source -in $tg.loaded_modules}
$tg|Add-Member -NotePropertyName commands `
-NotePropertyValue $commands
</code></pre>
<p>Remember the [yourObject].loaded_modules is mostly for information purposes, but if we implement this feature we should pay attention
to proper naming the modules, because of the <code class="highlighter-rouge">Where-Object</code> cmdlet.</p>
<h2 id="aliases">Aliases</h2>
<p>It is a good idea store our alias in profile, it’s syntax is <code class="highlighter-rouge">new-alias [alias name] [command name]</code> so for example we add the following line to the script</p>
<pre><code class="language-Powershell">new-alias gep get-executionpolicy
</code></pre>
<h2 id="drives">Drives</h2>
<p>We can define folders to behave like drives.
It’s also a good idea to move these commands in our profile.
For example the default windows user directory can be a fancy drive.</p>
<pre><code class="language-Powershell">New-PSDrive -Name "user" `
-PSProvider "FileSystem" `
-Root "c:\Users\undergrinder\"
</code></pre>
<p><img src="/assets/20180401/7.png" alt="" title="PSDrive in action" /></p>
<h2 id="logging-our-session">Logging our session</h2>
<p>Creating transcript is a handy method, I suggest everyone to include it in his/her profile.
Transcript is a text file log about your session, it stores what commands did you use, and they’s outputs.
Further information see the <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.host/start-transcript?view=powershell-6">official site</a></p>
<p>First we create a variable, that holds the name of our transcript, and add to our object:</p>
<pre><code class="language-Powershell">$sessionlog = $($tg.transcript + `
$env:userdomain +'-'+ `
$env:username +'-'+ `
(get-date -f 'yyyyMMddHHmmss')+'.log')
$tg|Add-Member -NotePropertyName current_transcript `
-NotePropertyValue $sessionlog
</code></pre>
<p>Append the next line to to end of our profile file:</p>
<pre><code class="language-Powershell">Start-Transcript $tg.sessionlog|out-null
</code></pre>
<p>After the pipe <em>out-null</em> makes sure that we don’t get verbose message about starting the script.</p>
<h2 id="godspeed">GodSpeed</h2>
<pre><code class="language-Powershell">Write-host `n'Have a nice session!' -foregroundcolor darkgreen
</code></pre>
<p>With the <em>`n</em> literal we ensure, that our message will start in new line.</p>
<h2 id="add-lucy">Add Lucy</h2>
<pre><code class="language-Powershell">write-host " ___.-'''-. "
write-host " ( (___,/\ \ "
write-host " \( |')' ) ) "
write-host " \) \=_/ ( "
write-host " ___ / _,' \ ) "
write-host " .' \|-(.(_|_ ; ( "
write-host " / //. (_\, | ) "
write-host " /`'---.._/ /.\_ ____..'| |_/ "
write-host " | /`'-._ / | '_| "
write-host " ` `;-''; | /,' "
write-host " `'.__/ ( \ "
write-host " '\/ "
write-host " "
</code></pre>
<h1 id="bring-all-together">Bring all together</h1>
<p><img src="/assets/20180401/8.png" alt="" title="AllTogether" /></p>
<h1 id="conclusion">Conclusion</h1>
<p>I’ve just guided you through the secret of profile creation, I’m sure you have already come up with ideas,
how can you expand your session.
If you have thoughts to share with me, leave a comment below.</p>
<p>You find an example profile script <a href="https://github.com/undergrinder/Scripts-examples/blob/master/PowerShell/profile-template.ps1">at my GitHub page</a>.</p>undergrinderFirst of all Happy Eastern to everyone! We got some day off because of it so we can dive deeper into a powershell customization project. Let’s see, how can we increase thr comfort factor using powershell (referred to as pshell) by tailor our profile.Www.ELCOME2018-03-31T00:00:00+02:002018-03-31T00:00:00+02:00https://blog.undergrinder.com/misc/2018/03/31/welcome<p><strong>Hey there!</strong></p>
<p>This page is my first attempt to spread my thoughts on the internet
(at the same time with other subdomain of <a href="https://dispatcher.undergrinder.com">undergrinder.com</a> - check it out).</p>
<p><img src="https://hu.gravatar.com/avatar/4a5da70980348332cb58df5fcdfd62fe?s=400" alt="avatar" title="Avatar" /></p>
<!--more-->
<p>The expected topics on the blog are…</p>
<ul>
<li>Databases</li>
<li>Script languages</li>
<li>Workflows</li>
<li>Reviews</li>
<li>MiniProjects</li>
</ul>
<p>I have multiple purpose with this site, among others…</p>
<ul>
<li>Learning new technologies</li>
<li>Learning the web by creating this site</li>
<li>Share my experiences</li>
</ul>
<h2 id="who-am-i">Who am I?</h2>
<p>My internet nickname is <em>undergrinder</em>. My first band name was Undergrind, that is founded by with my friends in high school, and it’s heritage is
two non-popular unkown hit, and my blogname.</p>
<p>I have have been working as BI developer (and meanwhile worked as PL/SQL developer too)
since 2014, and glad to say it, my work is my hobby. I like write scripts and create new things from scratch or
deal with existing tools.</p>
<p>This blog exist, because I want to try out <a href="https://jekyllrb.com">Jekyll</a>, and test it’s capabilities.</p>
<p>Beside computer things I’m interested in…</p>
<ul>
<li>Retro video games</li>
<li>Playing metal with two precious and at the same time infamous band.</li>
<li>Kettlebell</li>
<li>Creating lists, like this</li>
</ul>
<h1 id="conclusion">Conclusion</h1>
<p>I hope you’ll like my page, I try my best. :)</p>undergrinderHey there! This page is my first attempt to spread my thoughts on the internet (at the same time with other subdomain of undergrinder.com - check it out).