vardan.pro

Grand refactoring of SublimeText Nodejs plugin

2017-11-30T08:58:26+00:00

I’m very happy to announce the second version of Nodejs Sublime Text plugin. This is big release with a big refactorings and new features.

Intro

On November 2016 I’ve started to support (this is said very strongly), started to play around with SublimeText Nodejs plugin. The plugin was very small and has several bugs and unknown issue under the hood.

In those days I was playing around with Nodejs and favorite code editor was SublimeText, as always. I installed Nodejs plugin from packagecontrol.io and found several bugs as I mentioned earlier. I thought… Ohh, this is awful, take in count that Nodejs was a hype, and still the hype, but nowadays with a small pressure of Go. IMHO. 😇

In a such situation I directly go the GitHub repository of the project and looking for recent activity, to be sure that project is not abandoned. I saw that last commit was made to the project was in late 2013. Also, users left issues, but with no answer from the maintainer. Then, I thought… There is a two piece of technology which I love, and I always want to contribute to the Open Source. No sooner said than done.

The plan was simple. Fixing bugs which I will encounter with. Over time, I realized problem in fixing bugs, because of plugin architecture, missing of tests, debugging capabilities etc.

The plans is changing

As I mentioned earlier I was suffering without tests and debug capabilities. One bug fix could fork another bugs and issues. More over, some bugs could be fixed on one platform and still exists on another ones. Then I decided to make my life easier and makes me more happier.

What was done

A lot stuff was refactored and reconsidered. All changes logical could be divided in two groups: for user experience and for developer comfort.

Developer comfort

Lets start with a list of changes and little explanation for what, how and possible why it was done.

The whole code of plugin in earlier versions was located at one big file Nodejs.py. Will make decision to divide and re-locate the whole code to base into logical modules and move into lib folder. Now he have the following logical parts:
1. nodejs_base.py - base command classes for the rest of Sublime Text commands
2. nodejs_command_thread.py - classes for running underlying OS specific commands
3. nodejs_commands.py - core classes for the plugin functionality (node_run, node_drun and etc.)
4. nodejs_constants.py - several constants for PLUGIN_PATH and etc.
5. nodejs_debug.py - debugging functions
6. nodejs_nvm.py - class for detecting and working with NVM (Nodejs Version Manager)
Added debugging capabilities. Throughout of the code you call debug functions to output some of debug data to the Sublime Text console. The data will be printed only if file .debug_plugin exists in the plugin root directory. Simple remove the file to disable debug output.
Unit testing was introduced with help of great job done by Randy3k. This is a great module for writing acceptance tests for Sublime Text plugins. You can also use it to write simple plain unit tests for core parts of your plugin using plain old Python.
The Randy3k unit testing plugin also allow you to run tests in CI (continuous integration) services such as Travis Ci and Appveyor. Bang! I used it as well as tests. Now we can test plugin functionality on different platforms with a different versions of Nodejs.
Plugin now requires dependencies to be installed: sheellenv and newterm, which means after installing the plugin the Sublime Text will be restarted.

With the help of all the stuff listed above, we can develop new or bug fixing old functionality by enjoying.🕺

User experience

Ok. We got to the most interest part - user interaction with plugin. As in previous part I will list of changes and reasons for it.

It sad to say, but from now we only support Sublime Text 3. Sublime Text 2 support is deprecated, because of long waited Sublime Text 3 official release. It’s not beta now.
The most annoying (I think there was around 3-4 bugs related with it) incorrect auto complete. Sometimes it had replaced the part before dot: os.chdir(), chdir() replaced the os. part. Sometimes it had duplicated the part after a dot: http.createServer() duplicated the http part. The issue was solved by rewrite tools/doc_builder.js by adding Nodejs standard library modules names to the Nodejs.sublime-completions as a separate auto complete items.
Now, on the plugin loading, it detects if a Node Version Manager is installed and if it is, it gets current Node version from the NVM and use it by default instead of globally installed one, if any.
Also, on the plugin loading, it build completions list for a current version of Nodejs
On the plugin loading, it install all Nodejs dependencies need for plugin tools, located under tools folder: doc_builder.js and uglify_js.js. Before, you will need to install it manually.
At the core plugin, now, detects different versions of Nodejs. At the moment of release it is a Nodejs 6 branch and 8 branch aka LTS and current stable.
Plugin settings now opens in style of Sublime Text 3, by edit_settings command, default settings on the left side and user custom settings on the right side
From now, commands (‘node_drun’, ‘node_npm_install’ and etc.) could be run only when type or the buffer/file is type of source.js
OK. What is the most important tool of every developer? You, right! It’s a debugger.

Debugger

In previous versions of the plugin, Nodejs Debug (+ arguments) command would run only the current file with a debug argument passed to node executable. It was a useless. Since version 6.3.0 of Nodejs it support debugging using Chrome DevTools debugger. Read more at Paul Irish Medium blog.

Now, when you run Nodejs Debug (+ arguments) the node executable will run current file with flags --inspect=localhost:60123 --debug-brk for version 6 of Nodejs and --inspect-brk=localhost:60123 for version 8. As you can see, debugger would run on a separate port 60123 which is not correlate with the default Nodejs - 9229.

Then, you can connect to the debugger using steps instructed by the plugin:

    Debugger is successfully started at localhost:60123.
Now you can open Google Chrome and navigate to chrome://inspect.
Then click Open dedicated DevTools for Node. 
After click Add connection and add connection to localhost:60123

What need to be done

Recreate functionality of Nodejs cli debugger using Sublime Text 3 dependency - sublime-pexpect, which I contributed to packagecontrol.io recently.
Track Nodejs processes ran by the plugin and kill only that processes in _kill_node_processes(). For that, need to be done work by contributing another Python package to the Sublime package control - sublime-psutil.
Run current JS file in new terminal, instead of output to the Sublime Text output panel.

Script for guessing MySQL table charset and collation

2017-07-10T17:40:26+00:00

Intro

Sometimes we getting in a situation where have MySQL database where we have tables and even and columns in that tables in different charset/collations.

In such situations, we can “stick” in the problem where a DB has one charset/collation and a table has another charset/collation. On other hands, we can have tables where charset/collation match with DB’s charset, but data inserted in that tables in a third charset/collation, and it seems like corrupted.

In such situation, we can have the following: when we trying to select data from a table whose charset/collation match with the DB’s charset or not match but inserted in different charset/collation we get the:

+------+------------------------------+----------------------+
| id   | name                | surname                       |
+------+------------------------------+----------------------+
| 5100 | Ð                   | Ð¡Ð˜ÐÐ®ÐšÐžÐ’                 |
| 5234 | Ð¡Ð•Ð Ð“Ð•Ð™        | ÐœÐÐ§Ð˜ÐÐ¡ÐšÐ˜Ð™              |
+------+------------------------------+----------------------+

The data seems like it to be corrupted. To solve the problem if the data exactly corrupted or by mistake inserted in the wrong charset/collation, we must check the data against charset/collation supported by your version of MySQL. And then we can decide, the data really corrupted (if we can not find at least one right charset/collation combination with readable text) or to the data applied wrong charset/collation.

Preparation

Okey. First of all, we must find out which charset and collation support your installed MySQL version. For that, we go the terminal and:

mysql -u root -p'<your password>' -e 'show collation'

The next, we must get the output and convert to the following format: the first column of the output and the second column separated by the symbol ‘|’. For that purpose, I just copy-pasted output to the Sublime Text editor and with a help of multiline editing feature got the needed format. For example:

latin1_swedish_ci|latin1
latin1_bin|latin1
latin1_general_ci|latin1
latin2_general_ci|latin2
latin2_bin|latin2
ascii_general_ci|ascii
ascii_bin|ascii

After you have done converting, save it under you $HOME directory path with name mysql.collations.txt.

Code

Okey. We get to the fun part 😊 The following simple line of the Bash script, get the input file prepared in the previous step and iterating through it. On the each iteration it changes a connection charset and collation and select one row from your desired table and duplicate its output to a file in the current working directory with name mysql.codepage.guess.log.

for row in $(cat "$HOME/mysql.collations.txt"); do charset=$(echo $row | cut -f2 -d'|'); collate=$(echo $row | cut -f1 -d'|'); echo -e "\t\tCHECKING: $charset => $collate"; mysql -u root -p'<your password>' -e "use <your db name>; set names $charset collate $collate; select * from <your table name> limit 1;"; done | tee mysql.codepage.guess.log

Result

Here is presented a few lines of the file mysql.codepage.guess.log from my run (the outpur is cut for simplicity):

		CHECKING: latin1 => latin1_swedish_ci
id	name	surname	
5100	А	СИНЮКОВ	
		CHECKING: latin2 => latin2_bin
id	name	surname
5100	??	??????????????

That’s all. Here you can see that the right charset/collation for my table is latin1_swedish_ci. On the production, someone before me has set collation for the table to the utf8_general_ci but data was inserted in latin1_swedish_ci. Thank you for your time 🕺

Released a new bugfix version of the Sublime Text NodeJS plugin

2017-06-03T19:21:27+00:00

I’ve just released a new bugfix version of the Sublime Text Nodejs plugin. Should say, this is a first version, since that moment when I started porting the plugin to the version 3 of Sublime Text, when all features working out of the box either installed by the Package Control or manually.

List of bug fixes:

Fix issue-63. Now can run built-in tools, when the package is installed by Package Control. See more at GitHub
Fix issue-65. Now the package resolves correct path for built-in tools. See more at GitHub

Released a new version of the Sublime Text NodeJS plugin

2017-05-13T16:58:16+00:00

I’ve just released a new bugfix version of SublimeText-NodeJs package. A release notes for the new version v1.5.5:

Fix issue-60. Now the plugin respects system env var $PATH See more at GitHub
Fix issue-61. Autocompletion duplicates root object. See more at GitHub
Fix issue-62. Turning off shortcuts. See more at GitHub
Generated completions for NodeJS v7.10

Simple backups for non-tech users with udev

2015-10-30T06:00:28+00:00

Intro

We are using SharePoint 2010 Foundation SP2 as our EDM system. Every day we are making backups of our EDM system on external hard drive. A week ago our CEO has decided to has a second copy of the archives and keep it into a safe.

No sooner said that done!

Let's make criterias of future backup system:

it must be automatic
it can do everyone, even non-tech users
it must do ONLY onto specific external hard drive

So, what is a system doing? System consists of the following components: a udev rule and a bash script, which run when you insert the concrete external hard drive into destination host. Also, the system depends on the following system packages: cifs-utils, at, ntfs-3g.

Backups of our EDM system is lying on a Windows (SMB) share. When the external hard drive is inserted, the bash script will be runned and will create an `at` job. The `at` job at the specific time (by default 11pm on Saturday) will remove all content on the hard drive, then copy all files and folders from the Windows (SMB) share to the external hard drive.

Source Code

The source code is available at GitHub and distributed under MIT License.

Installation

As every backuping system, our system also need a some bit of configuration. First of all you need to install the system's required packages: ```bash sudo apt-get install git-core cifs-utils at ntfs-3g -y git clone https://github.com/varp/udevbackup.git && cd udevbackup ``` Second you need obtain your external hard drive serial number. To obtain serial number run: ```bash sudo udevadm info --query=all --attribute-walk --root /dev/sdb1 ``` The next phase of configuraion: ```bash HARD_SERIAL= SMB_DOMAIN= SMB_SHARE=<//server/share> SMB_USER= SMB_PASSWORD= make configure ``` </p>

To install just run the following in a terminal: ```bash sudo make install ```

Installation of TeamViewer on Ubuntu via SSH (Terminal)

2015-05-06T05:26:00+00:00

Intro

When I searched a solution to access remotely my Unity desktop on 14.04, I have faced with a little problem. It appears to be that all the solutions I found (xdrp, RealVNC, X11 forwarding) might only work with 2D desktop environments like as XFCE, Mate (the first two solutions) or can only forward “raw” X apps’ renders (the last solution). No one of it fit my needs.

The only solution that fits needs was the TeamViewer. It’s not hundred percent a UNIX solution, but it appears to be the only one solution that can give me access to Unity 3D session. Ok, I thought, let’s install TeamViewer! But, I was have only SSH access to my Ubuntu box.

Installing

If you will try to install TeamViewer 64bit deb you will be fail because in Ubuntu 14.04 (amd64) and above the lib32asound2 package was removed which on the TeamViewer 64bit depends. Though, we will install i386 package of TeamViwer. Enough words!

$ cd ~/Downloads && wget http://download.teamviewer.com/download/teamviewer_i386.deb
$ sudo dpkg –add-architecture i386
$ sudo dpkg -i ~/Downloads/teamviewer_i386.deb
$ sudo cp -v /opt/teamviewer/tv_bin/script/teamviewerd.sysv /etc/init.d/teamviewerd
$ sudo teamviewer daemon enable
$ sudo teamviewer daemon start
$ sudo update-rc.d teamviewerd defaults
$ sudo teamviewer --passwd <at least 8 characters>
$ sudo reboot

Reconnect to your Ubuntu box after it was rebooted and type in SSH terminal:

sudo teamviwer info

There you will see version of TeamViewer, ID of your TeamViewer to connect to and the TeamViewer daemon status.

That’s all. Grab your TeamViewer ID and remotely enjoy your Unity 3D session =).

Hacking SharePoint 2010 DB Performance

2015-02-08T20:44:00+00:00

Intro

We are using SharePoint Foundation 2010 SP2 as EDM system in our organization. We faced with a performance issue, after 3 years of using SharePoint 2010.

The Issue Context

Every employee in our organization has a pool of tasks to do. The pool of tasks is implemented as a SharePoint list. A task may has a dozen of executors. The executors leaving comments to the task overtime. Therefore, version history of the task (a list item) is growing and growing. On some tasks, we have about 2500 version history items. A SharePoint 2010 wasting about 150000 ms to retrieve these tasks.

Solving The Issue

WARNING! YOU CAN LOOSE SUPPORT ON YOUR COPY OF SHAREPOINT!

Open SQL Server Management Studio and run following script on a content database of your SharePoint instance:

USE [WSS_Content];
GO
create nonclustered index AllUserData_tpID on dbo.AllUserData ( tp_ID ASC );
create nonclustered index AllUserData_tp_ListId on dbo.AllUserData ([tp_ListId] ASC);
create nonclustered index AllUserData_tp_SiteId on dbo.AllUserData ([tp_SiteId] ASC);
create nonclustered index AllUserData_tp_RowOrdinal on dbo.AllUserData ([tp_RowOrdinal] ASC);
create nonclustered index AllUserData_tp_DeleteTransactionId on dbo.AllUserData ([tp_DeleteTransactionId] ASC);
create nonclustered index AllUserData_tp_Level on dbo.AllUserData ([tp_Level] ASC);
create nonclustered index AllUserData_tp_IsCurrentVersion on dbo.AllUserData ([tp_IsCurrentVersion] ASC );
GO

After that, I have achieved increasing of the DB performance up to 100 times.

Some usefull Linux command line shortcuts

2015-01-14T05:13:36+00:00

Snort 2.9.7.0 and Daq 2.0.4

2015-01-14T04:57:00+00:00

Intro

On past weekend I’ve packaged latest stable versions of snort and daq for Ubuntu Trusty. I’ve pushed the packages to my PPA. Enjoy and use it who it will be useful.

Upgrading

$ sudo service snort stop
$ sudo apt-add-repository ppa:vardan-pogosyan/stable
$ sudo apt-get update && sudo apt-get upgrade -y

If you are using a PulledPork, go to the PulledPork config file in /etc/pulledpork/pulledpork.conf by default and change snort_version option to 2.9.7.0. Then go to terminal and run:

$ sudo pulledpork.pl -c /etc/pulledpork/pulledpork.conf -l -m /etc/snort/sid-msg.map -P

The above need to update a snort dynamic rules lying in shared libraries under /usr/lib/snort_dynamicrules to version 2.9.7.0 otherwise snort won’t start. After upgrading the snort rules restart snort service.

If you are not using PulledPork for updating snort rules, you will need to do it manually.