My Computer Adventures

DBFImport

2018-10-31T00:00:00+00:00

A simple cross-platform tool to import old DBF files into SQL Server.

DBFImport

A simple cross-platform tool to import old DBF files into SQL Server.

DBF files were/are used by

dBase (Ashton Tate)
FoxBASE/FoxPro (Fox Software) / Visual FoxPro (Microsoft)
Clipper (Nantucket / Computer Associates)
…

Built on DotNet Core 2.1, works on Windows, Linux, Mac.

Usage

dotnet DBFImport.dll -p <dbfpath> -s <server> -d <database> [--codepage <cp>] [--nobulkcopy]
dotnet DBFImport.dll -p <dbfpath> -c <conn> [--codepage <cp>] [--nobulkcopy]

   -p, --path                Required. Path to DBF file(s)
   -s, --server              Required. SQL Server (and instance)
   -d, --database            Required. Database name
   -c, --connectionstring    Required. Database connection string
   --codepage                Code page for decoding text
   --nobulkcopy              Use much slower 'SQL Command' interface, instead of 'SQL BulkCopy'
   --help                    Display this help screen.
   --version                 Display version information.

The database must be an existing database. Every DBF file is imported into a database table with the same name (if the table exists, it is dropped first). A dot (‘.’) is displayed per 1000 records that are INSERTed in the database.

Examples

dotnet DBFImport.dll "c:\My DBF files\*.DBF" DEVSERVER\SQL2017 ImportedDbfFiles
#Imports DBF files:
dotnet DBFImport.dll --path "c:\Data\My DBF files\*.DBF" `
   --server DEVSERVER\SQL2017 --database ImportedDbfFiles
#Imports DBF files, and decode text using code page 1252:
dotnet DBFImport.dll --path "c:\Data\My DBF files\*.DBF" `
   --server DEVSERVER\SQL2017  --database ImportedDbfFiles --codepage 1252 
#Imports DBF files, and connect to SQL Server using connection string:
dotnet DBFImport.dll --path "c:\Data\My DBF files\*.DBF" `
   --connectionstring "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword"

Typical output

PS C:\> dotnet .\DBFImport.dll `
>>     --path 'c:\Data\My DBF files\*.DBF' --codepage 1252 `
>>     --server 'DEVSERVER\SQL2017' --database 'ImportedDbfFiles'
Processing c:\Data\My DBF files\ABALANS.DBF...
  LastUpdate:       16/08/2018
  Fields:           14
  Records:          5
  Importing:
  Inserted:         1
  MarkedAsDeleted:  4
  Duration:         00:00:00.9591436
Processing c:\Data\My DBF files\ANALYT.dbf...
  LastUpdate:       26/10/2018
  Fields:           22
  Records:          33496
  Importing:        ................................
  Inserted:         32402
  MarkedAsDeleted:  1094
  Duration:         00:00:00.4355719
...

Import finished.
Statistics:
  Records:          4869097
  Succeeded files:  714
  Failed files:     0
  Total Duration:   00:02:46.6250966

Download & installation

Requires dotnet Core 2.1 (available on Windows, Linux, Mac).
- Go to https://getdotnet.org, click “Download”, click “Download .net Core Runtime”.
- Or install using Chocolatey: choco install dotnetcore-runtime
Download & unzip the zip-file DBFImport.zip from https://github.com/WimObiwan/DBFImport/releases/latest

Typical speed characteristics

For what it’s worth, this is a real situation:

DBFImport.dll running on high-end laptop with SSD disk, Intel i7, 16 GB RAM.
DBFImport.dll compiled in Release mode, running on Windows 10 (1804), using dotnet Core 2.1.205 SDK.
Import of 714 local DBF files, total size 2493014358 bytes (2.32 GiB)
Containing 4868381 records (excluding those marked for deletion)
{1} Import into a local SQL Server 2017 Developer Edition.
{2} Import into a remote SQL Server 2016 Standard Edition (server specs unknown, virtualized, hosted in a datacenter)
In every test, the existing (filled) tabled was dropped and recreated.

	{1} Local SQL Server 2017	{2} Remote SQL Server 2016
SQL Command (`--nobulkcopy`)	16:11.421 (971 s) 4823 records/s 2.36 MiB/s	(*) 50.5 hours 27 records/s 13.39 KiB/s
SQL BulkCopy	2:05.705 (126 s) 38729 records/s 18.91 MiB/s	9:48.394 (588 s) 8274 records/s 4.04 MiB/s
Speedup of BulkCopy	7.73x	308.98x

(*) This test took ages, these numbers were obtained by extrapolation.

Resource utilization of dotnet.exe:

	{1} Local SQL Server 2017	{2} Remote SQL Server 2016
SQL Command (`--nobulkcopy`)	CPU: 4-8% (during 16 minutes) Memory: 12.2 MB	CPU: 0% Memory: 10.9 MB
SQL BulkCopy	CPU: 12% constantly (during 2 minutes) Memory: 12.1 MB	CPU: 1-4% (during 10 minutes) Memory: 11.7 MB

Impact of the option `--nobulkcopy`

When using BulkCopy, alle INSERTs are done in Batches of 10000 records. When an error happens, the current batch is rolled back, and the BuldCopy is aborted. All records INSERTed during previous batches will be present in the table.
When not using BulkCopy, every INSERT is done (separately using a prepared SQL Command), inside a single transaction. This means that when an error occurs, the complete transaction is rolled back, and no records will be present in the database.

Known issues & To do’s

Any feedback (issues or pull requests) is welcome!

Not alle datatypes from the xBase flavors are implemented.
Memo types (and DBT files) are not (yet) implemented.
Currently only SQL Server is supported.

Alternatives

There are alternatives:

Microsoft OLE DB Provider for Visual FoxPro 9.0 (VfpOleDB.dll)
Visual FoxPro ODBC Driver
Commercial products

For me these were insufficient because of:

incompatibilities between 32-bit drivers and 64-bit SQL Sever
instabilities of SQL Server when using some of the drivers in linked servers

References

SQL Server performance troubleshooting - Part 1

2018-10-26T00:00:00+00:00

SQL Server data/log file fragmentation can decrease performance. This post gives some fixes for file and index fragmentation.

There are 2 types of fragmentations:

Fragmentation of the data/log files on file system level.
- The cause of physical file fragmentation is mainly the auto-grow feature in SQL Server. This causes the data or log files to be extended in chunks whenever more space is needed. Using auto-grow is discouraged: it causes heavy fragmentation, and causes a performance hit every time the auto-grow is performed. I have even seen a yoyo-effect when SQL command execution time is set to a low value: an INSERT command causes an grow operation which takes a long time, causing the INSERT to time out, and causing the grow to be reverted. The INSERT was then retried forever by the application, every time with the same result.
- Disk fragmentation is only relevant for HDDs. There is a huge seek time penalty compared to sequential reads, which makes reads a lot more efficient when they can be done sequentially.
- Defragmenting SSDs is discouraged. There is no such seek time penalty. Moreover, defragmenting will decrease the lifetime of your SSDs. More information on StackOverflow/SuperUser.
Fragmentation of the pages inside the data/log.
- SQL Server uses pages (of 8KB), grouped in extents (of 64KB), to allocate space inside the files (cfr. Microsoft). Event when the files are not fragmented physically, the pages/extents will get fragmented over time.
- Pages of indexes should not be completely full, since this will decrease performance because the first insert (or key update) on the page will cause a page split. This is controlled by the Fill factor (cfr Brentozar).

Backup

Don’t forget to make a fresh backup before doing these operations.

The easiest way to do it is using SQL Server Management Studio:

Open the backup window:
1. Right click on your database
2. Open the Tasks submenu
3. Click Back Up…
Configure and start the backup:
1. Don’t forget to enable Copy-only backup. This is needed to prevent messing up with existing (transaction log & differential) backups, which would get corrupt when a new full backup is done outside the schedule.
2. Set a new location for the backup by removing the existing one, and adding a new one.
3. Click the OK button to start the backup.

Physical file fragmentation

(This doesn’t apply to SSDs. Only do physical file defragmentation on regular HDDs!)

There are several options to defragment this.

One way to do it is running a defragmentation of the complete disk from within *Windows Explorer. The advantage is that there are no extra tools needed. The drawback is that this defragments all files (not only the data/log files). If your data & log files are stored on different disks (which is a good practice), this needs to be repeated for every disk. Be aware that there can be multiple data files.
1. Open Windows Explorer and right click on the drive where the files are stored.
2. Click Properties to open the disk properties.
1. Go to the Tools tab
2. Click Optimize, and click Optimize again on the next window to start the defragmentation.
A last remark when using SSDs: The Optimize process on SSD disks doesn’t do a defragmentation, but just sends a trim to reclaim free blocks. So this doesn’t apply to this article on performance. More info on StackOverflow/Superuser
There is a way to run defragmentation on file level, which requires the SysInternals contig.exe tool.
Two ways to get it:
- By downloading it from the SysInternals website.
- Or by installing using Chocolatey:

    choco install sysinternals

Next you analyze individual data/log files by running:

  contig.exe -a '&lt;filepath&gt;'

Doing the actual defragmentation can be done by removing the -a:

  contig.exe '&lt;filepath&gt;'

In this example, the defragmentation has turned the 14 original fragments into 1 contiguous fragment. In a more realistic scenario which was executed at a customer, almost 9000 fragments were turned into 2 fragments.
The contig.exe tool doesn’t report any progress of the file defragmentation. But the progress can be monitored by running contig.exe using -a in another window, where you will see that the fragments are decreasing rapidly.
In theory, it is perfectly safe to kill a running contig.exe, by pressing Ctrl-C. The tool can be restarted later on, causing it to continue where it was stopped. It is unclear if contig.exe is as smart as the whole-disk Optimize tool mentioned above, with regard to SSDs. So I don’t recommend running it on SSDs.

Logical file fragmentation

In this topic, we are talking about index fragmentation. But be aware that (in the likely case that the table has a clustered index), a table is just a special kind of index.
A SQL Server index can be considered fragmented when it has more than 5% fragmentation. Tables smaller than 1000 pages will normally not benifit from defragmentation.

The most fragmented indexes can be found by running this query on your database: (to be repeated per database)

WITH fragmented_indexes AS
(
	SELECT 
		s.name [schema], t.name [table], i.name [index], ips.page_count, 
		ips.avg_fragmentation_in_percent, 
		ips.page_count * ips.avg_fragmentation_in_percent weight
	FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) ips
		JOIN sys.indexes i ON ips.object_id = i.object_id AND ips.index_id = i.index_id
		JOIN sys.tables t ON i.object_id = t.object_id
		JOIN sys.schemas s ON t.schema_id = s.schema_id
	WHERE page_count >= 1000 and avg_fragmentation_in_percent >= 5
)
SELECT * 
FROM fragmented_indexes
ORDER BY weight DESC;

Or if you have many indexes on many tables, the next query can be used to show a list per table:

WITH fragmented_indexes AS
(
	SELECT 
		s.name [schema], t.name [table], i.name [index], ips.page_count, 
		ips.avg_fragmentation_in_percent, 
		ips.page_count * ips.avg_fragmentation_in_percent weight
	FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) ips
		JOIN sys.indexes i ON ips.object_id = i.object_id AND ips.index_id = i.index_id
		JOIN sys.tables t ON i.object_id = t.object_id
		JOIN sys.schemas s ON t.schema_id = s.schema_id
	WHERE page_count >= 1000 and avg_fragmentation_in_percent >= 5
)
SELECT 
	[schema], [table], COUNT(1) indexes, SUM(page_count) page_count,  
	AVG(avg_fragmentation_in_percent) avg_fragmentation_in_percent, 
	SUM(weight) weight
FROM fragmented_indexes
GROUP BY [schema], [table]
ORDER BY weight DESC;

Defragmenting indexes: indexes can be Reorganized (recommended for low fragmentation, e.g. between 5% and 20%), or Rebuild (recommended for heavy fragmentation, e.g. more than 20%). Index Reorganization can be done on-line. Index Rebuilds can only be done on-line on SQL Server Enterprise Edition.

On SQL Server Standard Edition (or above), you should run a Maintenance plan to reorganize/rebuild indexes regularly, e.g. every week.
Alternatively, Ola Hallengren has maintenance scripts that can used even on SQL Server Express Edition where no SQL Server Agent is available. (Kohera)[https://kohera.be/blog/sql-server/providing-a-sql-server-express-edition-with-a-solid-maintenance-plan/] shows how to use the Windows Task manager as an alternative.
Indexes can be defragmentated manually, per table or index, by running REORGANIZE or REBUILD on indexes.
1. Expland the table entry in the Object Explorer.
2. Right click on the Indexes node (to run the operation on all indexes) or an individual index under the Indexes node (to run the operation on a single index).
3. Select Rebuild or Reorganize.
You can investigate the progress of a REORGANIZE operation by running the SQL script above. The index fragmentation will get lower in real-time while the operation is running.

pflogsumm

2018-09-29T00:00:00+00:00

pflogsumm is a great tool to generate a (daily) report on Postfix logs. SRS-rewrites invalidate the From-addresses, but the log files can be preprocessed as a workaround.

More information on pflogsumm can be found in this man-page

If you configured SRS on your Postfix mailserver, From-addresses get rewritten to

SRS0=HHH=TT=booking.com=mailing@foxinnovations.be

To allow correct reporting on From-addresses and From-domains by pflogsumm, you can replace the SRS-addresses by there original counterparts by running this script:

cat /var/log/mail.log.1 /var/log/mail.log \
	| sed 's/<SRS0=[^=]*=[^=]*=\([^=]*\)=\([^@]*\)@.*>/<\2@\1>/g' \
	| sed 's/<SRS1=[^=]*=[^=]*==[^=]*=[^=]*=\([^=]*\)=\([^@]*\)@.*>/<\2@\1>/g'

For example, to mail a report about yesterday, this script can be run:

cat /var/log/mail.log.1 /var/log/mail.log \
	| sed 's/<SRS0=[^=]*=[^=]*=\([^=]*\)=\([^@]*\)@.*>/<\2@\1>/g' \
	| sed 's/<SRS1=[^=]*=[^=]*==[^=]*=[^=]*=\([^=]*\)=\([^@]*\)@.*>/<\2@\1>/g' \
	| /usr/sbin/pflogsumm -d yesterday > /tmp/mail 2>&1
cat /tmp/mail | mail -aFrom:myserver@example.com -s "[Mon] Mail stats" me@example.com

You can call this script from your crontab, e.g. every day, a few minutes after midnight.

Zabbix SIP server monitoring

2018-08-01T00:00:00+00:00

Zabbix can be used to monitor availability and up-time of a SIP server or phone.

A template can be found on GitHub: Link

The template uses an external script to run the check. You need to put the contents of the externalscripts subfolder under your Zabbix external scripts folder (on my Zabbix installation on Debian, this is /usr/lib/zabbix/externalscripts/)
Import the template in the Zabbix user interface:
- Go to Configuration, Templates
- In the top-right corner, click on Import
- Select the template XML file and click Import

SQL Server performance testing

2018-04-04T00:00:00+00:00

With your databases stored on SSD-disks on fast development machines, it becomes difficult to reproduce performance problems that occur on customer machines. This post gives some tricks to allow troubleshoot performance problems, while still using your fast development machine.

Store your database data file on an external USB disk

Before you detache the USB drive, you should:

In SQL Server Management Studio: Bring the database offline (or detach the database)
In Windows: Prepare the USB drive to detach (as you should always do with USB-disks)

If you happen to forget one of these, it can happen that the database gets corrupted. When this happens, you can correct the problems by running a DBCC CHECKDB as in single-user mode:

ALTER DATABASE [MyDatabase-USBDISK]
SET SINGLE_USER
WITH ROLLBACK IMMEDIATE;
GO
 
DBCC CHECKDB;
GO
 
ALTER DATABASE [MyDatabase-USBDISK]
SET MULTI_USER;
GO

Prevent caching of query data in memory

A slow USB disk will make the query slow once, but after that, the query will fast because the data is cached in (much faster) memory. To prevent this, you can run this query:

WHILE 1=1 -- repeat forever

BEGIN
    -- clear query cache

    DBCC FREEPROCCACHE;
    -- clear data cache

    DBCC DROPCLEANBUFFERS;
    -- wait 3s

    WAITFOR DELAY '00:00:03'
END
GO

This query will run forever unless an error happens (e.g. connection issue) or the query is canceled by the user.

RDP credential caching

2018-03-18T00:00:00+00:00

In Microsoft’s RDP client, you can enable a checkbox to save your password. Usually, this doesn’t work, because by default delegating the credentials isn’t allowed by the Group Policy. This post explains how to enable this.

When you connect to a Terminal Server, you can get this error message when using saved credentials: Your credentials didn’t work. The credentials that were used to connect to [some server] did not work. Please enter new credentials.

Follow these steps to enable delegating credentials:

Open the Group Policy Editor as administrator:
One way to do this is:
- Press the Windows-key, and type Powershell.
- Powershell will be found: right click on the result, and select Run as administrator.
- In the Powershell prompt, type gpedit.msc,and press Enter.
Update the settings in the Group Policy Editor:
- In the left-side tree pane, navigate to:
  - Local Computer Policy
  - Computer Configuration
  - Administrative Templates
  - System
  - Credentials Delegation
- In the right pane, double click on:
  - Allow delegating saved credentials with NTLM-only server authentication
- Select the radio button Enabled:
- Click on Show… beside Add servers to the list.
- Add a new entry TERMSRV/*to enable delegation for all servers:
- Repeat the same for Allow delegating saved credentials.
Apply the Group Policy with the new changes:
- In the Powershell prompt, run the command gpupdate.

Android Auto, and how to use it in non-supported countries

2018-03-10T00:00:00+00:00

Android Auto is a great application on your Android phone, that integrates seamlessly with all types of new cars. In many countries it isn’t available (yet?), but luckily - with some simple steps - it can be installed and used.

The feature is amazing: you connect your phone using USB to your car, and some of the applications become available on the screen of your car media system. On Opel Insignia cars, the feature is called Projection.

First the main screen shows:
«TODO IMAGE»

After connecting the phone, the Projection icon changes into the Android Auto logo:
«TODO IMAGE»

After touching the icon, you get a literally some kind of projection of the Android Auto application on your car media screen:
«TODO IMAGE»

Other applications like Waze and Spotify integrate in Android Auto:
«TODO IMAGE»
«TODO IMAGE»

According to the Android Auto site, the application is only available in these countries: Argentina, Australia, Austria, Bolivia, Brazil, Canada, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, France, Germany, Guatemala, India, Ireland, Italy, Japan, Mexico, New Zealand, Panama, Paraguay, Peru, Puerto Rico, Russia, Spain, Switzerland, United Kingdom, United States, Uruguay and Venezuela. So if you live in Belgium or the Netherlands, you can’t install the application from Google Play. It doesn’t even show up in the search results, unless you are in one of the supported countries, and you are not logged in with your Google Account. When you search on the APK ID and try to install it on your phone, you get the error This item isn't available in your country..

The list of supported cars and car media is impressive: Abarth, Acura, Alfa Romeo, Alpine, Audi, Buick, Cadillac, Chevrolet, Chrysler, Citroën, CVTE, Dodge, DS, Fiat, Ford, Genesis, GMC, Holden, Honda, Hyundai, JBL, Jeep, JVC, Karma, Kenwood, KIA, Lamborghini, The Lincoln Company, Macrom, Mahindra, Maruti/Suzuki, Maserati, Mercedes-Benz, Mitsubishi Motors, Nissan, Opel, Panasonic, Peugeot, Pioneer, RAM, Renault, Seat, Škoda, Smart, Sony, SsangYong, Subaru, Suzuki, Tata Motors, Vauxhall, Volkswagen and Volvo. Others have already announced the support of Android Auto: Bentley, Blaupunkt, Borgward, Caska, Infiniti, Iveco, Jaguar, Koenigsegg, Lada, Land Rover, Mazda, MMauto, Renault Samsung Motors, SmartAuto, UniMax and Zenec.

Android Auto runs on Android 5.0+ (Lollipop, Marshmallow, Nougat, or Oreo) and you need a data connection.

To force installing Android Auto on your phone in a non-supported country, you can manually install the APK.

First you need to allow installations from sources other than the Google Play store:

Open the Settings on the phone
Tap on the Security item
Make sure the checkbox “Unknown sources” (Allow installations from sources other than the Play Store) is enabled
After you finished the steps below, you should restore the setting.

Then you can install the APK manually: on your phone, download & run the APK.

One of the sites you can use is APK Mirror.
Open the website on your phone.
At the bottom, select the most recent version.
Select the correct architecture (most likely arm64)
Tap the button DOWNLOAD APK
After the download, open the APK file, which will install the Android Auto application.
When finished, you can restore the setting to allow “Unknown sources”.

Getting started with VS Code using C# and .NET Core on Ubuntu

2016-10-24T00:00:00+00:00

A tutorial on how to compile and run C# application on Ubuntu with Visual Studio Code and .NET Core.

Steps:

Download & install Visual Studio Code from this link.
For Ubuntu, download the DEB file and install by double clicking on it.
Enable the C# plugin
- Ctrl-Shift-X
- Search for C# (provider Microsoft)
- Enable the plugin and restart Visual Studio Code.

Install .NET Core (as documented on this link)

# For Ubuntu 16.04, but also works on Ubuntu 16.10.
sudo sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet-release/ xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893
sudo apt-get update
sudo apt-get install dotnet-dev-1.0.0-preview2-003131

Microsoft development tools build numbers - Visual Studio 2015 Update 3

2016-10-07T00:00:00+00:00

It is quite a challenge to understand the versions and build numbers of the Microsoft build chain. This post tries to list them up…

Click here to see the complete list of versions

Visual Studio 2015 build numbers

Date	Name	IDE Version	C++ Compiler
2016-09-14	Visual Studio 2015 Update 3 “d”	14.0.25431.01	19.00.24213.1
2016-08-02	Visual Studio 2015 Update 3 “c”	14.0.25425.01
2016-07-20	Visual Studio 2015 Update 3 “b”	14.0.25424.00
2016-07-12	Visual Studio 2015 Update 3 “a”	14.0.25422.01
2016-06-27	Visual Studio 2015 Update 3	14.0.25420.01	19.00.24210

Outlook stuck on splash screen

2016-06-07T00:00:00+00:00

After running Microsoft Outlook for years without problems, it won’t start anymore after a reboot.

This problem happened with Microsoft Outlook 2016, connected to an Office 365 online account. But I assume it is rather unrelated to the version.

After a reboot, I launched Outlook, and this was the result:

The splash screen was shown infinitely, while indicating the status


_Repairing_ the OST file with the tools from the Microsoft website didn't help.

This was the only thing that helped:

* kill outlook
* restart outlook in _safe_ mode (this succeeds immediately)
* close outlook
* restart outlook (which now also succeeds immediately)

A few PowerShell commands to do this:

```bash
# Kill outlook (or click the 'x' in the right/top corner of the splash screen)
(Get-Process outlook).Kill()

# Find the location of outlook.exe
$reg = 'HKLM:\SOFTWARE\Classes\Outlook.Application\CLSID'
$outlook_clsid = (Get-ItemProperty $reg).'(default)'
$reg = "HKLM:\SOFTWARE\Classes\CLSID\$outlook_clsid\LocalServer32"
$outlook_exe = (Get-ItemProperty $reg).'(default)'

# Start outlook in safe mode
. $outlook_exe /safe

My Computer Adventures

DBFImport

DBFImport

Usage

Examples

Typical output

Download & installation

Typical speed characteristics

Impact of the option --nobulkcopy

Known issues & To do’s

Alternatives

References

SQL Server performance troubleshooting - Part 1

Backup

Physical file fragmentation

Logical file fragmentation

pflogsumm

Zabbix SIP server monitoring

SQL Server performance testing

Store your database data file on an external USB disk

Prevent caching of query data in memory

RDP credential caching

Android Auto, and how to use it in non-supported countries

Getting started with VS Code using C# and .NET Core on Ubuntu

Microsoft development tools build numbers - Visual Studio 2015 Update 3

Visual Studio 2015 build numbers

Outlook stuck on splash screen

Impact of the option `--nobulkcopy`