normalize int data

Question

Hello everyone I have a couple of integers and I want to find a simple and effective way to normalize them into range [0 1]. Any ideas and code refer? thanks in advance George

George George · Answer

Cool Alan! Your math skills are so cool! regards George

Alan · Answer

Yes 59 will be normalized to (approximately) 0.5. I always start by looking for the midpoint of the range to get an idea of how much skew the normalization is introducing. If you consider the position when i == min then temp is zero and the normalized value is: Exp(Log(Lower + 1)) - 1 which is: Lower + 1 - 1 = Lower Similarly for i == max then temp is unity and the normalized value is: Exp(Log(Lower + 1 + Higher + 1 - Lower - 1)) - 1 which reduces to: Exp(Log(Higher + 1)) - 1 which is: Higher + 1 - 1 = Higher and it's now easy to see that any intermediate input will be normalized to a value in the interval [Lower Higher].

George George · Answer

Cool Alan! You are making Finance as well as C#!! Two more comments 1. 0.5 corresponding to an integer of 59 does it mean input 59 will be normalized to 0.5? 2. how do you prove return Math.Exp(Math.Log(Lower+1) + temp * (Math.Log(Higher+1) - Math.Log(Lower+1))) - 1; will be in range [lower higher]? thanks in advance George

Alan · Answer

I'm not aware of any 'official' normalization methods though it's a long time since I did any serious mathematics and I would think in any case that it depends in which field you're working and the probability distribution of the variable you're dealing with. I just did a search on logarithmic interpolation and came up with this link: http://books.google.co.uk/books?id=tt3RLXsagagC&pg=PA74&lpg=PA74&dq=logarithmic+interpolation&source=web&ots=LfPy-51BHH&sig=X8R07MZRLPRyrDgtp9zNm2Yu4TU&hl=en This suggests yet another approach: public double LogNormalize2(int i) { if (i if (i >= Max) return Higher; double temp = (double)(i - Min)/(Range - 1); return Math.Exp(Math.Log(Lower+1) + temp * (Math.Log(Higher+1) - Math.Log(Lower+1))) - 1; } Once again I had to add one before taking logs to avoid NaN values though I then subtracted one to balance it out. The results were pretty good with 0.5 corresponding to an integer of 59 so this time the results are skewed towards the lower half of the range though not too much. This should be suitable for a lot of more distributions than the last approach.

George George · Answer

Cool Alan! I am wondering whether there is any official logarithm or linear based normalization methods (e.g. MD5 has official implementation in RFC) or there are just some experience based or best practices based implementation like you did in this post? regards George

Alan · Answer

The best way to test that is to run through all possible integer values and then total and average the normalized results for each approach: using System; class Program { static void Main() { int min = 1; int max = 100; Normalization norm = new Normalization(min max); double total = 0.0; double total2 = 0.0; for(int i = min; i { total += norm.LogNormalize(i); total2 += norm.LogNormalize2(i); } Console.WriteLine(Average value when adding one is : {0:F3} total/100.00); Console.WriteLine(Average value when not adding one is : {0:F3} total2/100.00); Console.ReadLine(); } } public class Normalization { public readonly int Min Max Range; public readonly double Lower Higher Interval; public Normalization(int minint max double lower double higher) { Min = min; Max = max; Range = max - min + 1; Lower = lower; Higher = higher; Interval = higher - lower; } public Normalization(int min int max): this(min max 0 1) { } public double Normalize(int i) { if (i if (i >= Max) return Higher; return Lower + (double)(i - Min)/(Range - 1) * Interval; } // add one to the logs public double LogNormalize(int i) { if (i if (i >= Max) return Higher; double temp = Math.Log(i - Min + 1)/Math.Log(Max - Min + 1); return Lower + temp * Interval; } // don't add one to the logs public double LogNormalize2(int i) { if (i if (i >= Max) return Higher; double temp = Math.Log(i - Min)/Math.Log(Max - Min); return Lower + temp * Interval; } } The results were 0.790 when adding one and 0.782 when not so I think the answer is that it doesn't make much difference. However if your actual range of numbers is less than 100 then you may want to run this code using the actual range to make sure the difference is still acceptable.

George George · Answer

Thanks Alan I want to analyze whether adding 1 will impact the result of logarithm based normalization algorithm but do not have enough knowledge. Do you have any ideas? Does it impact or not or not so much? regards George

Alan · Answer

Hi George The reason why I defined Range as Max - Min + 1 is because that's what the range actually is when we include both endpoints and because I was exposing it as a public readonly field where the caller might expect such a value to be returned. In the logarithmic code I added 1 to the numerator to prevent Math.Log(i - Min) becoming 0 when i was equal to Min + 1 which would have caused the method to return a value of 0 for both i == Min and i == Min +1. Having added it to the numerator I then had to add it to the denominator as well to prevent a value of 1 being returned for both i == Max - 1 and i == Max. Although it seems a bit crude I believe it's a standard technique when dealing with logarithmic interpolation.

George George · Answer

Cool Alan! Two comments 1. In your original linear code why not change to the following code to save time to minus 1 each time? Range = max - min; return Lower + (double)(i - Min)/(Range ) * Interval; 2. Why in your logarithm code you add 1 to both Math.Log(i - Min + 1) and Math.Log(Max - Min + 1)? regards George

Alan · Answer

Firstly I've just noticed that there's a mistake in my normalization formula (even though it works correctly when the interval starts at 0) in that 'Lower' needs to be added on to the result. The corrected method is therefore: public double Normalize(int i) { if (i if (i >= Max) return Higher; return Lower + (double)(i - Min)/(Range - 1) * Interval; } I'd imagine that with log based normalization you use logarithmic rather than linear interpolation to determine where the integer value will map within the interval. That would suggest something like: public double LogNormalize(int i) { if (i if (i >= Max) return Higher; double temp = Math.Log(i - Min + 1)/Math.Log(Max - Min + 1); return Lower + temp * Interval; } However this isn't going to be suitable for all distributions as it skews the normalized values to the upper half of the interval. In the example program a value of 0.5 corresponds to an integer of 10 and so 90% of the range will be above 0.5.

George George · Answer

Cool I heard there is also a logarithm based normalization approach do you know how to implement in this way? regards George

Alan · Answer

Something like this perhaps: using System; class Program { static void Main() { Normalization norm = new Normalization(1 100); double result = norm.Normalize(50); Console.WriteLine(result); Console.ReadLine(); } } public class Normalization { public readonly int Min Max Range; public readonly double Lower Higher Interval; public Normalization(int minint max double lower double higher) { Min = min; Max = max; Range = max - min + 1; Lower = lower; Higher = higher; Interval = higher - lower; } public Normalization(int min int max): this(min max 0 1) { } public double Normalize(int i) { if (i if (i >= Max) return Higher; return (double)(i - Min)/(Range - 1) * Interval; } }