So, what do I do while my class is working on labs? Well, if I’m not answer emails from the Navy, I’m working on some side projects. Right now, I’m working on a project to grab meta data via geographical coordinates. When I develop code, I brake the objective down into manageable parts. I then look at each part and ask myself if I know how to do it. For each part that I do not know how, I start writing code to discover the path to success.
Well, while working on a bit of code, I needed to increment my value for latitude by .05. Something odd happened so I wrote out a small bit of code to focus in on the issue. Here is the code.
$Increment = .05
$Start = 0
$Value2 = $Start
While ($Value2 -lt 10)
{
Write-host $Value2
$Value2 += $Increment
}
The data type of $Value2 is Double. Everything starts out fine.
0
0.05
0.1
0.15
0.2
Until…
3.45
3.5
3.55
3.6
3.65
3.69999999999999
3.74999999999999
3.79999999999999
3.84999999999999
Why System.Double start incrementing in .049999999999999 and then back to .05 later on, I do not know. OK, let’s force the variable to be System.Single
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5000001
0.5500001
0.6000001
0.6500001
OK, now it increments by 0.050001 and then back to .05.
Luckily, I found a data type that does not loose it’s accuracy under these conditions, System.Decimal.
$Increment = .05
$Start = 0
[Decimal]$Value4 = $Start
While ($Value4 -lt 10)
{
Write-host $Value4
$Value4 += $Increment
}
$Value4 | GM
Below is a sample of the desired output.
0
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
I took a look at System.Double on MSDN. According to Microsoft, System.Double coheres to the standard for binary floating-point arithmetic. (See the Remarks section). Since some fractional values cannot be represented precisely. Look at System.Decimal. In the Remarks section, you will see that it dose not have a round off error. At least not at the precision level that I need.
I now need to decide between the smaller memory footprint of Single or Double data types as compared to the much larger storage requirement of Decimal. It looks like that I will be computing the Longitude and Latitude values in Decimal, But using the –as operator in PowerShell to convert them into Single for storage since I do not need to go beyond 2 decimal places.
[Decimal]$Decimal = 5.5
$Single = $Decimal -as [Single]
$Single
The result will be a Single value that will have the precision that I need with a smaller memory footprint.
Comments