What is the difference between phpFox CDN and CDN Service apps? + some questions about the CDN

1)Answer
phpFox CDN =  You can host CDN files by your self such as video photos  etc...

CDN Service = third party paid service it is good for geolocation

2)From information you gave, What you need is phpFox CDN

3)Put 2 files and create folder  /file/ 

and change setting in admin CP

Please read me thread

Bug & fix for phpFox CDN
https://community.phpfox.com/forum/thread/57516/bug-fix-for-phpfox-cdn-4-6-1/

4)You cannot
Files will be random to assign each CDN server.

5)If a CDN  full  I think it will show some error and  fail to save file and it not auto to save to another CDN.  
User have to re upload again it will save to another CDN

1) Some problem if you run 2 CDN on VPS

If you site getting larger and more hits  or if you video post get viral  like  10,000  people load your site at the same time

a CDN on VPS cannot be handle you will need load balancer in front of CDNs

2) Load balacer in front of CDN

Behide load balancer you can setup multiple CDN servers 

load balancer -- > cdn.yourdomain.com  --->  5 CDN Servers

3) if you site getting larger you can setup multiple load balancer  with multiple CDN on each load balancer

Load balancer -- > cdn.yourdomain.com   --->  5 CDN Servers

Load balancer -- > cdn2.yourdomain.com  --->  5 CDN Servers

Load balancer -- > cdn3.yourdomain.com  --->  5 CDN Servers

Hi vktechnology!

Thank you for detailed answers to my questions!

You've helped me understand this issue very much!

Regarding the points:
1. OK, now I understand the difference between these applications.
However, you mentioned that with more users online, you may have a problem if I have to say 2 CDN servers on 2 different VPS servers (this is just an example).

You mentioned that I would need a balancer in front of CDNs in this case.

Ok, it's logical and I understand it.

Just how to set this balancer in front of CDNs?
In the phpfox settings can not be set?

3. If I only have to place these two files on some VPS server that will serve as a CDN server and change the settings in the Admin CP in the phpFox CDN application, it looks like it is relatively simple. I understand that I can place these files on different servers? Is there a limitation here?

Thank you also for providing a link to the thread where you described problems in the phpFox CDN application.

I have seen this thread before, as I was looking for answers to my questions. I will take this into account when configuring the CDN.

Thanks!

P.S.
It's a bit weird that phpfox has not yet fixed this bug in the phpFox CDN application.

4. That's not good. In the default phpfox configuration, if I do not use the CDN server, all video files are in one folder, so I can set up a hyperlink for this folder.

The point is that the video files are on a separate disk.
It follows that when using the CDN server, all the files are together, which is a problem for me (I store the pictures on an SSD drive, while the video files I wanted to store on a slower hdd).

5. Ok, I thought so.
I have a question at this point.
Suppose I have three CDN servers on three different VPS servers (one VPS server - one CDN server).

Let's say that one of the CDN servers will be full. In this case, can I somehow transfer some of the files to another CDN server?

There will be no problem? All I need to do is move files, will there be something to be set up?

6. I have one more question. Can you advise me something?

At the beginning I wanted to create one CDN server and add more CDN servers as needed. I will use the phpFox CDN application.
Do you have any tips on how to do it wisely, so that there are no problems in the future? And what if the space on the first CDN server ends and the other will have a lot of space?

Will there always be an error if the user adds a photo, and phpfox will want to add this image to the first (full) CDN server? You wrote that the user will have to add the photo again and phpfox will re-route it to the next CDN server.
Only I would like to avoid such situations that such mistakes will not be repeated in the future. As needed, I would like to balance the occupied space between the CDN servers from time to time (see item 5).
It is known that I will not buy 10 CDN servers at the very beginning. I would like to start with one or two and if needed, add more.

Thank you again for your answer vktechnology!

Regards,

kerstar

Answer

1) How to setup load balancer ?

You need HAproxy

https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts

HAproxy download from here

http://www.haproxy.org/

You can learn how to setup load balancer for normal website not CDN

after that you will understand the concept and you configure for CDN later on

3)You can put 2 files and change setting in file upload to any server.

I don't think there any limitation on how many CDN you can have .

4) CDN should not set by default on PhpFox  becasue when people start install phpfox it not easy.

I think you don't need to separate disk between video and pictures

4.1) If you place video in once CDN and If you site get alots of hits on video your site will slow and video use alots of bandwidth

What about if people upload and watching video  your HDD will have alot IO Wait on slow HDD and site slow becasue you put all video in one place.

Buttle neck on video CDN

4.2)For CDN what you need is larg web space and have multiple CDN to reduce load

setup Large space SATA HDD on Raid 10 is good enought for a CDN

5) If one CDN full

You can do many mothod

You be setup script in crontab to check HDD space and if 85% it send to your email

5.1) Copy to new HDD after that you can prepare to add new HDD and copy (rsync) old data to new HDD 

stop CDN server and remonth new HDD

5.2) Before it full you copy files to new CDN server (This case if your CDN behide load balancer )

and on your load balancer rerout your fulled CDN to new CDN server IP

This method less down time

6) Sugestion if you plan to have larg site

1) You need to design simple and clean system, You have well understanding on the system you setup

it will help you and your team easy to maintain and easy to grow your servers.

2) You need to learn HAproxy load balancing and setup load balacing for your phpfox main site  (not CDN) it not hard you can do it.

3) You can start with 2 CDN servers no need to setup load balacing for CDN

4) After you site getting lrager you can setup load balancer for your CDN

Sugesstion for future  version of CDN

1. Feature CDN WEIGHT  on adminCP 

It will help to balance space on old and new CDN  instead of random write to CDN

We can put value of weight on each CDN  ( Like we can put weight for each server  on HAproxy )

Weight is % of write file to each CDN server

Weight =0  that  CDN will read only no files will be writtenit on that CDN it good when storage is full or nearly full

Why we need this feature ?

1)If we setup CDN for long time old CDNs will have alots of  used space

2) When we add new CDN this CDN have alots of free space

3) We will need  to write alots files to new CDN then we can set WEIGHT to high value in order to balance space on each CDN

For eaxmple we can set weight on each CDN

CDN Weight

cdn1  = 0

cdn2  =  30

cdn3  =  70

Mean if write new files to CDN  70% of new files to be written will  go to cdn3.

cdn1 will read only because it full of space.

Wow! A very detailed answer to my questions. Thank you very much! Your answers helped me to understand CDN in phpfox.

Regarding the points:

1. Ok, I read about it HAProxy and everything seems logical.

However, I have a few more questions here:

You recommend using the load balancer for CDN servers, and also the main page. OK, everything is clear.

It is known that the load balancer will be useful for the phpfox server, which has many online users, so as to distribute traffic to several servers.

Only here is the problem: How to maintain the consistency of data between web servers on which phpfox (not CDN) is installed?

The idea is that the cache memory and other settings that are contained in it are the same on all servers.
Will there be a problem with phpfox licensing (installation is allowed on only one server)?

Is it not worth using the load balancer for the web server and focus only on using the load balancer for the CDN servers?

I think to myself that a strong single dedicated server that supports www traffic (the one on which phpfox is installed) should handle traffic 10,000 - 15,000 online, am I right?

And in that case I would focus only on using load balancer for CDN servers.
Or, are not I right?

4. You mentioned that you think there is no need to separate the discs between photos and video.
I will not agree here. Photos that add users in phpfox are compressed and weigh relatively little compared to the original photos that the user adds. However, unfortunately, phpfox does not support video compression that users add. This is a big downside, because if a user adds a 50 megabyte movie, the server will have 50 megabytes of space instead of a few megabytes of compression.

That's why I decided that the photos will be kept on the SSD system, and the movies on the hdd drive (hdd disk is slower, but much cheaper).

I am currently looking for a person who will help me implement the compression of the added videos by the user (lowering the bitrate, setting the maximum frame size, reducing the audio quality, etc.). This is to be done, but unfortunately I can not set it myself.

5. Here you are right, but at the same time you did not understand me.

Let's say I have 2 CDN servers.

Server number 1 has 250 GB of capacity, server number 2 also has 250 GB of capacity.

Your solutions rely on increasing the size of the disk. Ok, let's say that I am running out of space on these two CDN servers and I am adding a larger 500 GB disk for each of these servers, and then transferring the data to them.

So we have this situation:

CDN Server number 1 and server number 2 now have 500 GB of storage.

But what if, once again, I will run out of space on these disks again and I will not be able to increase their size?

A good solution would be to add another CDN server.

Ok, but since the CDN server number 1 and the CDN server number 2 have 90% of the space used and the new number 3 CDN server has a lot of free space, then the server number 1 and server number 2 will overflow (because as you mentioned, files are assigned randomly).

And here is my question:
Will I be able to transfer some of the files from the CDN server number 1 and 2 to the number 3 CDN server, so as to balance the amount of space on individual CDN servers?

All I need to do is transfer these files to server number 3?
Is it impossible?
If it can not be done, how can it be remedied?

You mentioned Feature CDN WEIGHT on adminCP.

It's a very good solution and it sounds logical.
But unfortunately he is not in phpfox ... How can you solve the problem I described in point 5?

Thank you again for sharing your knowledge. You've helped me so much. I think that other people will also learn from this thread.

Best wishes!

Answers

1) Sharring data between servers by using NFS and conect locally via very fast nework. 

1.1) About License Install on load balacing no problem with license on other software I think same as phpfox.
Public will see as only one domain and only one installed.

1.2) A strong regular dedicated server  host  webserver php + mysql  cannot handle phpfox 10 -15 K cucurent users.  very far from true.

If you want to large phpfox site on self hosted you have to use load balancing.

first step put load balancing on main phpfox site.

4) Video compression  will caused high load on server processors and take log time.

Most people will upload video from mobile phone that already compress to mp4 size already small.

You can set limit size and file type of video

5) HDD Space

If you need to run large site  CDN only 500 GB is not enoght and SSD is expensive

For example on one CDN  you need  4TB  Sata HDD x 4  on raid 10 = 8 TB  

5.1) You can tranfer to  3th CDN but  not easy to do for example you will have to down your site inorder to copy files and I think have to change some value in databse table to tell  which file on which CDN

That is complicate and too dangerous, if you do somethign wrong in database your site will down for long time

When you are on large system what you need is to do something in simple way not complicate.

to start not to use too small CDN space you will get this problem

The simple way is "CDN WEIGHT feature"

if your site grow large if phpfox not have this feature you can  hide some developer to do it.

Hello vktechnology

Thanks for explaining.

I have a question and a problem.

At point number 1 you did not understand what I mean.
The use of load balancer for phpfox main site is that we copy the contents of the current phpfox main site server to 2-3 other servers?

And in the HAproxy configuration, we point to these 2-3 servers?

I understand what the load balancer function is.

My question is whether the current server on which the phpfox is to copy to 2-3 other servers and HAproxy to point to these servers?
That's all?

If I have three servers: phpfox main site server 1, phpfox main site server 2 and phpfox main site server 3, how to set it to have the same data on each of the three servers? Do you understand what I mean?
If load balancer redirects the user to, for example, phpfox main site server 1 and another user to phpfox main site server 2, how will the files between these servers be synchronized? What if the user who connects to phpfox main site server 1 adds the attachment. How will another user using another phpfox main site server see this attachment if it is on another server? What does it suit? All three phpfox main site servers will share one database (here there is no problem with synchronization, because one database is for these three servers), but after all phpfox main site server 1 will not see any changes that will happen in phpfox main site server 2 or phpfox main site server 3.

You know what I mean?

The issue I noticed today in phpfox.

I've properly configured the CDN server.
If a user adds a photo, it is visible in this folder:
cdn1.mydomain.com/file/pic/photo/2019/07

Everything is fine here.

But...I noticed that this image was also added to the folder:
mydomain.com/PF.Base/file/pic/photo/2019/07/

This is a bug, because the photo is unnecessarily duplicated, and undermines the sense of using the CDN server.

Do you know how to fix it? Or did I miss something?

Regards,

kerstar

I didn't read the entire thread as I was skimming but  I noticed you mentioned the cdn keeping files in both places.  If I remember correctly and you might want to wait for phpfox to respond but there is a setting called:

Keep Files In Server

Set this to Yes to keep original files on the server.

I think they have this option in case you decide not to use a cdn anymore you are able to reconnect everything but may be more difficult if you have both files deleted.  Like I said it been a few years so I can't remember the exact reason.

I submitted this question to the phpfox support team and I am waiting for a response.

Apps >Core >Settings

Keep Files In Server  = NO

This will not keep original files on server and keep only on CDN

Hi kerstar 

Is your site already live or just still under development ?

1) Assume you already setup HAproxy configuration for 3 servers.

1.1 Copy phpfox on server 1 to server 2 and 3.

1.2 if you need to share same files for all servers just use NFS and map directory (You will have to read what is NFS, how to use and setup)

1.3 Connect all 3 servers to same MySQL.

1.4 All 3 servers use CDN server for attachments. When user upload to any server files will go to CDN server.

When users on any servers  they willl see same data

2) Convert your PhpFox Database to INNODB 

Configure  all phpfox DB into RAM

INNODB engine type is more stable than MyISAM and row leve locking.

It helps to prevent table locking when user update table. (table locked = site freez)

(You will have to read about InnoDB.

You can learn by setup simple harpoxy share data by using NFS between server, connect to same mysql and it will help you to understand more.

Hi vktechnology!

My site is still in development.
I have been working on the site since December 2017 and only now I could start work with the release of phpfox 4.7.3, because only in my opinion was it devoid of significant issues that I reported.

Earlier versions were not suitable for creating a social network.

In principle, work on the development of the page is at the finish, I still have a few things to set up.

Thank you again for your tips. It helped me a lot and I will definitely use your advice when setting up your phpfox (as much as I can).

As for duplicate files, you were right. It was enough to disable this option in AdminCP (I did not know before that there is such an option).

Regards,

kerstar